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EDITOR’S INTRODUCTION 


Waotty within the past two decades, and largely since 
1909, a vast change in educational procedure has been 
effected in this country as the result of the development of 
scientific methods as applied to the measurement of both 
the children to be taught and the instruction to be given. 
It is no exaggeration to say that the introduction of these 
new working tools has done more than any preceding addi- 
tion to educational theory or practice to transform education 
from an art into a science, with a well-organized body of 
scientific procedure to be mastered in preparation for the 
work. So fundamental has been the change wrought by 
the introduction of these new tools that school supervision 
is fast being changed from guesswork into scientific accu- 
racy, and the apprentice method of preparing for the work 
is giving way to professional and technical preparation in 
schools organized for that special purpose. 

Where formerly we had only a technique of classroom 
practice, organized in the form of a rather minute method- 
ology, to enable us to estimate the quality and the effective- 
ness of classroom work, we now have tests and scales and 
norms which are rapidly being standardized and made ac- 
curate. Where formerly we had only the teacher’s guess as 
to the children who were bright and average and dull, we 
now have graded and standardized group and individual 
intelligence scales for use in determining the native ability 
the pupils possess for mastering the work of the school. 
Similar scales are now being developed for use in guiding 
the older pupils in the choice of a life-career. Where pre- 
viously we graded and promoted pupils largely on the basis 
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of their chronological age, we to-day can determine their 
mental age with accuracy and know that it is the more im- 
portant basis of the two to use. Where formerly we ex- 
pected pupils to master the course of study somewhat 
equally, and graded and advanced pupils rather evenly, we 
to-day understand how to calculate the achievement quo- 
tient for each pupil in the school and know that it is the real 
measure of ability to make school progress. Where for- 
merly we used only simple arithmetic in tabulating results, 
and spoke in terms of the simple arithmetic average, we 
to-day have a well-organized body of mathematical knowl- 
edge and statistical procedure to apply to the analysis and 
interpretation of the results, and these results we express in 
terms of median, mode, frequency-distribution, percentiles, 
probable error, and degree of correlation. 

So rapidly have these new procedures been evolved, due 
to the large number of vigorous young workers who have 
been attracted to this new phase of educational activity, 
that probably nothing to-day serves so well to differentiate 
the newer scientifically trained type of school supervisor 
from the older successful-practitioner type of educational 
worker as does the ability to use intelligence scales, to 
measure and diagnose by means of standardized educational 
tests, and to apply the new statistical methods to the analy- 
sis of the results obtained. The mastery of these new tools 
has to-day become almost essential in the equipment of any 
man or woman who now aspires to educational leadership 
as a school principal or a supervisor of instruction. 

So rapidly, too, has this new knowledge been developed, 
organized, put into textbooks, and disseminated, that we 
have now reached the point where an advanced textbook 
on tests and measurements, dealing with the fundamental 
theory lying back of the construction, use, and interpreta- 
tion of educational tests, is needed for the use of students 
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who have first become familiar with them through the 
means of elementary textbooks or schoolroom practice. 
This the present volume in this series of textbooks, written 
by one who has done much to develop and standardize and 
apply educational tests, aims to supply. It ought to prove 
very useful as a textbook in a second or advanced course in 
Educational Tests and Measurements in Teachers’ Colleges 
and Schools of Education, and it can be read with much 
profit by superintendents and principals of schools who have 
previously become somewhat familiar with tests and test- 
ing procedure. 
Extiwoop P. CuBBERLEY 
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PREFACE 


As we have advanced in the construction and use of stand- 
ardized objective tests it has become clear that we should 
endeavor to refine our measuring instruments and our use 
of them. In order to accomplish this it is necessary that we 
become conscious of the assumptions upon which our meas- 
urements are based, and that we come to understand more 
fully the principles of the construction and use of educa- 
tional tests. We need to become familiar with the tech- 
nique for making critical studies of educational tests, and 
for evaluating them with reference to their functions. 

This volume is the outgrowth of an attempt by the au- 
thor to give, to advanced college and graduate students, a 
course which would equip them so that they would be able 
to make critical studies of educational tests and to form in- 
telligent judgments with reference to their usefulness. Since 
the manuscript was begun, several years ago, it has under- 
gone numerous revisions. As there was no satisfactory text 
available, several of the chapters have been typewritten 
and made available for intensive study by the members of 
the author’s classes. During the school year of 1921-22 
the first eight chapters were studied and criticized by a class 
of eight graduate students. 

Although the volume is the direct outgrowth of the au- 
thor’s attempts to instruct college students, he hopes that 
it will afford a means of giving to superintendents and others 
who direct the use of educational tests in our public schools 
a point of view, as well as acquainting them with certain 
fundamental principles which seem necessary in order to 
secure the maximum returns from the investments that are 
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being made in this field. The number of available educa- 
tional tests has multiplied until it is necessary for one who 
is formulating a testing program to make a selection of the 
tests to be used. This selection should be made intelli- 
gently. After the tests are selected they should be used to 
increase the efficiency of the school. It is the author’s hope 
that this volume will prove helpful in assisting those work- 
ing with educational tests to accomplish these ends. 

Two chapters of statistical methods are included. In one 
sense they are not an integral part of the volume. They 
are intended to be used as a reference by those readers who 
are not sufficiently acquainted with statistical methods to 
understand the procedures described in connection with the 
construction and evaluation of educational tests. For a 
more complete treatment of statistical methods the reader 
should consult authoritative texts in this field. 

An author is naturally indebted to many persons. In 
the present case the author feels that his greatest indebt- 
edness is to his students, particularly those enrolled in his 
class in “Theory of Educational Measurements” during 
the first semester of the school year of 1921-22. He is 
also indebted to his colleague, Professor E. H. Cameron, 
who has read the manuscript of several of the chapters. 
He desires to acknowledge his indebtedness to Dean F. J. 
Kelly and Professor J. C. DeVoss for their permission to 
utilize certain materials from Educational Tests and Meas- 
urements, of which they are co-authors. They also have 
read certain portions of the present manuscript. The writer 
is indebted to a large number of other workers in this field. 
The number is too great to mention here, and appropriate 
acknowledgments are made in the body of the text. 


Water Scorr Monror 
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AN INTRODUCTION 
TO THE THEORY OF 
EDUCATIONAL MEASUREMENTS 


CHAPTER I 


THE BEGINNINGS OF STANDARDIZED OBJECTIVE 
TESTS 


Origin of standardized objective tests. Although the 
achievements of pupils have been measured in some form 
since schools were organized, the construction of standard- 
ized objective tests for this purpose is a comparatively recent 
development. These measuring instruments have two dis- 
tinguishing characteristics. First, they tend to be objective, 
which means that the measures which they yield are rela- 
tively independent of the person who makes them. In other 
words, different examiners will obtain approximately the 
same results when measuring the abilities of the same pupils 
under the same conditions. Second, these tests are stand- 
ardized, so that norms are available for interpreting the 
measures which they yield. For example, if the measure of 
a given achievement of a fifth-grade pupil is 41, this magni- 
tude can be compared with the norm or standard for the fifth 
grade. By doing so we can ascertain whether this pupil 
is above standard, just up to standard, or below standard 
with respect to this particular achievement. 

The idea of standardized objective tests is relatively sim- 
ple. It is likely that the idea had been comprehended, in 
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part, by many teachers before it was developed by recent 
workers. Professor E. L. Thorndike! has called attention to 
a plan for measuring the achievements of pupils which was 
used by an English schoolmaster as early as 1864. The in- 
strument, called a “‘scale book,” was crude, but it included 
the germ of many of the ideas which are incorporated in our 
present-day educational tests. Sample performances, rep- 
resenting various degrees of excellence, were collected and 
arranged in the form of a scale similar to our present hand- 
writing scales. Such “scale books” were constructed for 
several school subjects, and they appear to have been used 
in the school where the idea originated. However, the 
work of this English schoolmaster, Reverend George Fisher, 
appears to be isolated from the development of educational 
tests in this country. In this connection, it should be noted 
that the construction of many of our present-day measuring 
instruments was not possible until appropriate statistical 
methods had been worked out. In this field Sir Francis 
Galton was the pioneer. The beginning of standardized 
objective tests in this country dates from the work of Dr. 
J. M. Rice, in 1894-95. 

The measurement of general intelligence, or capacity to 
learn, was developed separately from the measurement of 
the achievements of pupils. Many attempts were made 
to “measure the mind”’ by indirect methods. Phrenology 
represented one attempt which attracted much attention. 
Physiognomy was another. Still later, certain physical 
measurements were proposed by some physiologists as a 
means of learning about a child’s mind. Closely connected 
with the physical measurements were those of reaction time 
and other mental functions. It, however, remained for 


1See Journal of Educational Psychology, vol. 4, pp. 51. A quotation 
from The Museum, a Quarterly Magazine of Education, Literature, and Sci- 
ence, vol. 111, 1864, is reproduced as a communication. 
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Binet, in 1905, to propose the method which is now gener- 
ally used. His work marks the beginning of our present 
measurement of general intelligence. Group intelligence 
tests are a very recent development. 

Two groups of workers contributed to the early develop- 
ment. The early development of standardized objective 
tests for measuring the achievements of pupils was contrib- 
uted to by workers in two fields, psychology and school ad- 
ministration. In studying the functions of the human mind, 
the psychologists frequently devised tests in school subjects. 
These tests were more or less crude, but, generally, they were 
objective. Standards were seldom determined, since they 
were of little value to the psychologists. The school admin- 
istrators were interested in questions of courses of study, 
school organization, methods of instruction, etc. Until a 
quarter of a century ago, such questions were dealt with on a 
controversial basis. The side which had the best “debaters” 
won. More recently, educators have attempted to answer 
these questions in a scientific manner and, in doing this, ob- 
jective tests have frequently been devised. 

The development of standardized objective tests is so re- 
cent, and so many workers have contributed to the move- 
ment, that it is difficult to select the persons and the events 
which deserve recognition. It is an interesting fact that all 
persons who have made major contributions are living at the 
time this is written (1922). However, a few persons and 
events appear to be distinctive. A number of these are men- 
tioned below. 

J. M. Rice, the pioneer, 1894-95. In America the first 
person whose work has been given a place in the develop- 
ment of standardized objective tests is Dr. J. M. Rice. His 
tests were not standardized, or at least he made no use of 
them as such. Since the test papers were rated under his 
immediate supervision, and in one investigation the test was 
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given by him in person, the objectivity of the tests was not 
an issue. Rice’s contribution to educational measurement 
was, rather, in the use of tests in certain school subjects for 
answering questions concerning the best course of study, 
time allotment, methods of instruction, and the like. In 
doing this he focused attention upon the idea of measuring 
the achievements of school children. 

Rice’s first investigation (1894-95) was in the field of spell- 
ing. His first test consisted of a list of fifty words. This was 
sent to a number of school superintendents, of whom twenty 
responded, sending in the returns from more than sixteen 
thousand school children. He constructed a second test, in 
sentence form. This test was given under Rice’s personal 
supervision, and the scoring was checked by his assistants. 
Among the conclusions which he drew from the results, Rice 
stated that pupils who had spent forty minutes a day for 
eight years in studying spelling did not appear to possess any 
greater ability to spell than those who had devoted only ten 
minutes a day to the subject. When he reported his inves- 
tigation, the educators of that day, with few exceptions, vig- 
orously attacked the implied thesis that it is possible to meas- 
ure the results of teaching spelling by ascertaining how well 
the pupils can spell certain words. This was done in spite 
of the fact that teachers had been and still were attempting 
to measure the results of their instruction in spelling by 
means of written tests and by other crude devices. A prin- 
ciple that was generally accepted in practice was rejected 
when utilized as a means of answering a disputed question 
concerning the optimal time to be devoted to the teaching of 
spelling. It was claimed that the object of teaching spelling 
“was not to teach pupils to spell, but to develop their 
minds.” Later Rice gave tests in arithmetic and language. 

E. L. Thorndike, 1903, 1904, 1909-10. Professor E. L. 
Thorndike has made many notable contributions to the de- 
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velopment of standardized objective tests. He also has con- 
tributed indirectly to the movement through the work of 
his students. At the time of the first of Rice’s work, Thorn- 
dike was a student of statistical methods. He was also 
deeply interested in the field of psychology. This combina- 
tion of interests began to bear fruit as early as 1903. In this 
year, in collaboration with Fox, he published a study enti- 
tled Relation between the Different Abilities in the Study of 
Arithmetic: In studying this relation, a number of tests on 
the operations of arithmetic were used. The motivation of 
this contribution was psychological rather than administra- 
tive. In this respect, it differs from the work of Rice, whose 
motive was largely that of the school administrator. 

In 1904 Thorndike published the first edition of his Men- 
tal and Social Measurements. In addition to an account of 
statistical procedure, this volume contained many of the 
principles upon which the construction of our present tests is 
based. It was revised in 1913, but the revision consists, 
primarily, of adding concrete illustrations of the principles. 
This book is yet an important source of information for 
workers in this field, and for a number of years was essen- 
tially the only source. 

In December, 1909, Professor Thorndike presented his 
handwriting scale before the meeting of Section L of the 
American Association for the Advancement of Science. The 
report of the derivation of this scale was published in the 
Teachers College Record the following March. ‘The ap- 
pearance of this scale was an important event in the devel- 
opment of educational tests. This scale is a device which 
makes possible the description of the general merit of hand- 
writing.in quantitative terms. This characteristic of a pu- 
pil’s handwriting is essentially qualitative but, by means of 
Thorndike’s Handwriting Scale, it can be described in quan- 


1Columbia University Contributions to Philosophy, Psychology, and 
Education, pp. 32-40 (February, 1903). 
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titative terms. By the use of the theorem that differences in 
quality which are equally often noted by competent observ- 
ers are equal, he secured a set of specimens of handwriting 
which differed from each other by known amounts of qual- 
ity. These specimens were arranged in order of increasing 
merit to form the scale. The quality of a sample of a pupil’s 
handwriting is measured by matching it with the division of 
the scale which it most nearly resembles in general merit. 
The appearance of this scale is significant also because it 
was constructed without special reference to a study which 
called for the use of a measuring instrument. Although 
Thorndike was not indifferent to the usefulness of his hand- 
writing scale, he was primarily interested in the technique 
of its construction. 

The principles underlying the construction of Thorndike’s 
Handwriting Scale were applied to the field of English com- 
position by Hillegas, in 1912. Since that time they have 
been used with some modification and elaboration by a num- 
ber of other workers.! 

C. W. Stone, 1908. The work of Rice inspired C. W. 
Stone, then a student under Thorndike, to undertake the 
study of two questions in the field of arithmetic: 

(1) What is the nature of the product of the first six years of 

arithmetic work? 

(2) What is the relation between distinctive procedures in arith- 

metic work and resulting abilities? 

In the work of Stone we have a combination of the inter- 
ests of the psychologist and of the school administrator. 
The former is due to Stone’s contact as a student with Thorn- 
dike, and the latter to the inspiration received from the 
work of Rice. 

In the study of the above questions Stone devised two 
tests in the field of arithmetic, one on the fundamentals, and 

1 See Chapter VI for an account of the method of scale construction. 
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the other on reasoning. These tests were objective, in that 
printed directions were provided for giving them and for the 
scoring of the test papers. These directions were sufficiently 
detailed to insure that different workers would obtain ap- 
proximately the same results in applying the tests to the same 
pupils. The tests, as used by Stone, were not standardized. 
Since the completion of the original study the reasoning test 
has been standardized, and used extensively in school surveys 
and by teachers. In the construction of his tests, Stone was 
more scientific than Rice. Weights were determined for ex- 
amples which obviously differed in difficulty, so that the 
units of the scale would be more nearly equal. 

S. A. Courtis, 1909. S.A. Courtis codperated with Stone 
by giving his arithmetic tests. Stone’s study was confined 
to the sixth grade. Courtis conceived the idea of giving the 
tests in all grades, including the high school. This he did 
during the school year of 1907-08. He states that he was in- 
terested in measuring the growth of pupils in arithmetic, and 
in establishing norms of attainment for the several grades. 
He found the Stone tests unsatisfactory for this purpose, 
and during the following year he devised a group of arithmetic 
tests which he called Series A. These were made available 
for use in September, 1909. The series consisted of eight 
tests. There was one test on the fundamental combina- | 
tions or tables in each operation. The fifth test called for 
the copying of figures, and Test No. 7 included examples 
from the four operations. Two tests were devoted to reason- 
ing or the solving of problems. A great deal of care was 
exercised in the construction of the tests, and in the formu- 
lation of the directions and record blanks for their adminis- 
tration. A time limit was set, so that measures of the rate 
of work were secured. The scoring was made objective, 
and the tests were standardized. 

The purpose of Courtis in devising this series of tests was 
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different from that of earlier workers in the field. His inter- 
ests were neither those of the psychologist nor those of the 
school administrator but, rather, those of the instructor. 
He thought of the tests as instruments to aid the teacher in 
instructing pupils. They were to do this by measuring prog- 
ress of the pupils, and the established norms were to be used 
by the teacher as objectives. The comparison of the meas- 
ures of achievement with the norms furnished a diagnosis of 
the pupils with respect to their educational needs. 

The extensive use of these tests in schools convinced 
Courtis that they were unsatisfactory in a number of respects. 
For this reason he devised and made available, during the 
year 1913-14, a new group of tests called Series B. It is this 
group of tests that is generally referred to as the “Courtis 
Standard Research Tests in Arithmetic.” The series con- 
sists of four tests, one on each of the fundamental operations 
with integers. The examples are relatively “long” and 
within each test all are the same size. In the addition test 
each example consists of three columns of figures of nine fig- 
ures each. The tests measure both rate and accuracy. 

L. P. Ayres, 1912, 1915. L. P. Ayres contributed to the 
development of educational measurements by the construc- 
tion of his handwriting scale (three-slant edition) in 1912, 
and his spelling scale in 1915. He has, also, contributed to 
the movement through numerous addresses. The Ayres 
Handwriting Scale (three-slant edition) represents an ingen- 
ious attempt to secure an objective index of the quality of 
handwriting. Legibility was adopted as the criterion of 
quality. Samples of pupils’ handwriting were read, under 
controlled conditions, and an index of legibility was calculated 
from the average rate of reading. This method of scale con- 
struction has not been followed by other makers of hand- 
writing scales because investigation revealed that Ayres’s 
scale is not distinctly superior to Thorndike’s, and the 
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method of construction used by the latter is much simpler. 
Ayres’s most significant contribution is his spelling scale. 
In it he emphasized the importance of determining the edu- 
cational objectives as a prerequisite of test construction. 

B. R. Buckingham, 1913. In 1913 B. R. Buckingham, a 
student under Thorndike, published the account of the deri- 
vation of his spelling scale. This work is largely theoret- 
ical and the scale has not been widely used. It is, however, 
significant historically because it represents a new type of 
measuring instrument. Words were selected and their dif- 
ficulty was determined on the basis of the percent of correct 
spellings by pupils in the various school grades. These words 
were then arranged in the order of their difficulty. This 
procedure produced a measuring instrument which began 
with words which practically all pupils could spell, and in- 
creased in difficulty until, at the other end of the scale, rel- 
atively few pupils were able to spell the words correctly. 
The idea underlying this type of measuring instrument is 

that the ability of a pupil may be measured in terms of the 
level of difficulty which he is able to reach on the scale, 

This principle of test construction has been followed by a 
number of workers in this field, notably, Woody in the oper- 
ations of arithmetic, Hotz in algebra, Henmon and Brown in 
Latin, Trabue in language, and Van Wagenen in history. 

Binet (1905), 1908. All of the events up to this point 
have been in connection with the development of instru- 
ments for measuring the achievements of pupils in school 
subjects. Because of the intimate relation of general intelli- 
gence to achievement, and because, as we shall show later, 
there is need for combining measures of intelligence and 
achievement, it is appropriate to mention briefly the devel- 
opment of general intelligence tests. 

In 1904 Binet, an eminent French psychologist, was ap- 
pointed by the French Minister of Public Instruction, as a 
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member of a commission for the organization of classes for 
subnormal children. This brought Binet face to face with 
the problem of devising some means for determining what 
children were subnormal. In theendeavor to solve this prob- 
lem he devised, in 1905, in collaboration with Simon, the 
group of tests which have become known as the “ Binet Gen- 
eral Intelligence Tests.””’ They were revised by Binet in 
1908. Binet’s most significant contribution was the use of 
age-norms in interpreting the performances of pupils in terms 
of mental age. 

The work of Binet attracted the attention of American 
psychologists, and his tests have been revised and improved 
by a number of our psychologists, notably, Goddard and 
Terman. The Stanford Revision by Terman is, perhaps, 
the most widely known and used. 

Group intelligence tests (1918). The Binet tests must be 
given to pupils individually. Since frequently more than an 
hour is required to administer them, it is obvious that their 
use is necessarily limited. Prior to the entry of the United 
States into the World War, in the spring of 1917, a number 
of psychologists had been experimenting with tests that 
might be given to pupils in groups. A. S. Otis, a student 
under Terman, had completed the derivation of such a 
test. When it was decided to introduce psychological testing 
into the United States Army, he generously placed his results 
at the disposal of the committee which had this work in 
charge. The derivation and extensive use of group intelli- 
gence tests for psychological examining in the United States 
Army is probably the most important event in connection 
with this type of test. Since the cessation of hostilities 
there have appeared a very large number of group intelli- 
gence tests, which are of the same general type as that orig- 
inated by Otis and adopted for use in the United States 
Army. ' 
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Other events. Many other events might be mentioned. 
In fact, the number of recent events is so great that one is 
unable to evaluate them properly. A great deal of ingenu- 
ity has been exhibited by the makers of silent reading tests. 
Notable among these are E. L. Thorndike, who has contrib- 
uted the Thorndike Scale Alpha II for measuring the under- 
standing of sentences, and the Thorndike Visual Vocabulary 
Test; and F. J. Kelly, who devised the Kansas Silent Read- 
ing Test. The latter event is particularly important be- 
cause of the simplicity of the test and its consequent wide- 
spread use. A very recent development, which at the pres- 
ent time appears to be of major importance, is the introduc- 
tion of derived scores, and the comparison of measures of 
achievement with measures of general intelligence so as to 
obtain achievement quotients. These two topics will be 
discussed in detail in later chapters. 

Studies of the accuracy of the school marks. The devel- 
opment of educational tests has been stimulated by a num- 
ber of events not having to do with their construction. Not- 
able among these are studies of the accuracy of school marks. 
These investigations revealed that school marks were highly 
subjective, and hence inaccurate. Perhaps the most impor- 
tant event in this connection was the publication, in 1914, 
of a monograph by F. J. Kelly, entitled, Teachers’ Marks.* 
In this monograph the author summarized the existing stud- 
ies of school marks, and added the results of a number of 
new investigations. This accumulation of evidence left no 
doubt that teachers’ marks were inaccurate, and the demon- 
stration of this fact created the need for instruments which 
would yield more accurate measures of achievements. 

School surveys. The survey movement has sustained a 
twofold relation to the development of educational tests. 


1 Certain of these studies will be described in Chapter II. 
2 Teachers College Contributions to Education, No. 66. 
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School surveys stimulated derivation and use of educational 
tests. In turn, the present type of school survey has been 
made possible by the derivation of instruments for measur- 
jng the abilities of pupils. The study of a school system by 
2, committee appointed for that purpose is not a new activity. 
One writer mentions the appointment of an educational com- 
mission of the City of Chicago, in 1894, as one of the ear- 
liest of these endeavors. The use of the word “Survey” in 
connection with the study of a school system appears to 
date from a “Survey ” of Pittsburgh, in 1907. Following 
this date school surveys shortly became common events. 
The inquiry into the public schools of New York City, in 
1911-12, is notable because it is the first survey in which 
educational tests were used to measure the achievement of 
‘pupils as a means of evaluating the efficiency of a school 
system. S. A. Courtis was appointed as a member of the 
‘survey commission, and gave Series A of his arithmetic 
‘tests to over thirty thousand children. Since this date there 
‘-have been few school surveys in which educational tests have 
not been used. In a number of school surveys, those making 
the study of the school system felt that the existing tests 
were unsatisfactory, and devised new instruments to meet 
their needs. Notable illustrations of this are the Cleveland 
Survey Arithmetic Tests, the Willing Composition Scale, 
and the Nassau County Supplement to the Hillegas Scale. 
Research organizations. The development of educational 
measurements has been greatly facilitated by the establish- 
ment of research organizations. These have been of two 
types. One has been organized in school systems, primarily 
for the purpose of directing and stimulating the application 
of educational tests. Sometimes the organization has been 
evolved out of the activities carried on by some division of 
the superintendent’s office. In other cases it has represented 
the creation of an essentially new department placed, gen- 
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erally, under the control and direction of the superintendent. 
One of the first of these organizations was in New Orleans, 
in 1912, but the department has since been discontinued. 
The same year a department was established in Rochester, 
New York. In 1913 a Bureau of Research and Reference 
was established in New York City. In the following year 
similar organizations were established in Kansas City, Mis- 
souri, and Boston, Massachusetts. 

The other type of research organization has been estab- 
lished by state educational institutions or state departments 
of public instruction. The Bureau of Measurement and 
Efficiency, at the University of Oklahoma, was established in 
1913. This appears to have been the first of such organiza- 
tions. A number of others were established very soon after. 
The Bureau of Coéperative Research at Indiana Univer- 
sity, the Bureau of Educational Measurements and Stand- 
ards at Kansas State Normal School, Educational Service, 
Extension Division at the State University of Iowa, and the 
Bureau of Educational Measurements at the University of 
Nebraska were established in 1914. The State Depart- 
ment of Public Instruction of Wisconsin established a simi- 
lar organization in 1915. More recently, organizations of 
this general type have been established in a large number of 
other state educational institutions, and recognition has been 
given to the work by the department of public instruction in 
a number of states. 

This type of research organization has been very fifinail 
tial in popularizing the use of educational tests. Through 
them, school superintendents and teachers have been made 
acquainted with the tests, and frequently they have been 
trained in the use of the tests. Service has also been ren- 
dered by making the tests easily accessible, and by standard- 
izing tests devised by other workers. In many instances 
those directing research organizations have devised new tests. 
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QUESTIONS AND TOPICS FOR INVESTIGATION 


1. What were the motives that actuated the pioneers in educational 
measurements? 

2. What reasons can you give for the attack that was made upon Rice’s 
work? 

3. How do you explain the fact that the test movement has been carried 
on almost entirely by young men? 

4, Trace the development of standardized objective tests for the differ- 
ent school subjects. 

5. How does Ayers’s Handwriting Scale differ from the one devised by 
Thorndike? 

6. What contribution has Courtis made in this field? 
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CHAPTER II 


NATURE AND PROCESS OF EDUCATIONAL 
MEASUREMENTS 


Two kinds of description. We apply two kinds of descrip- 
tion to physical objects. When a thing is described in terms 
of its color, texture, form, beauty, and so forth, we have 
qualitative description. When we tell how large a thing is, 
the description is quantitative. Take, for example, a table. 
The kind of wood of which it is made, the color of its finish, 
and its shape are terms which may be used in its qualitative 
description. A statement of its weight, height, dimensions 
of its top, and other information which tells how large it is 
describes it quantitatively. 

To measure a thing is simply to describe it in quantitative 
terms. Whenever we determine how much or the amount 
of any characteristic of an object, we have measured that 
object with respect to this characteristic. 

Scales used for quantitative description. All quantitative 
description requires a scale of quantity.1_ The most common 
type of scale is one which is formed by laying off the unit of 
measurement from a starting-point. This starting-point is 
called the zero point, and usually means not any of the char- 
acteristic which is named in the unit. This scale may be 
extended indefinitely to meet the needs of measurement. 
An object is measured or described by determining how far 
it extends along this quantitative scale from the zero point. 
Thus, when the height of a tree is described as being twenty- 
three feet, this means that the height extends along the scale 


1 By a liberal interpretation “quantitative description” may be taken to 
include description in terms of large, very large, heavy, tall, etc. It is used 
in a more restricted sense in this book. 
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of linear measure a distance of twenty-three units (feet) from 
the zero point on the ground. 

It is necessary that the unit which is used in forming the 
scale be of the same kind of magnitude as that which we are 
attempting to measure. For example, we cannot use the 
scale of which the square foot is the unit to measure the 
weight of an object. Furthermore, the magnitude which we 
are attempting to measure must be consistent throughout. 
These requirements are so obvious in the case of physical 
measurements that it may appear unnecessary to mention 
them. Attention is called to them here because similar re- 
quirements must be satisfied in the measurement of mental 
traits. 

Another type of scale which may be used in quantitative 
description is one consisting of a series of objects of a given 
type arranged in order of increasing magnitude. The scale 
objects may be numbered in order, starting with the one of 
least magnitude, or they may be given other convenient nu- 
merical designations. When this has been done, the numer- 
ical name of a scale object defines a definite magnitude. 
Quantitative description by means of such a scale consists 
in estimating which of the scale objects the object being de- 
scribed most nearly resembles in magnitude. The position 
of the scale object in the scale describes the other quantita- 
tively. If the scale objects differ from each other by known 
amounts of the magnitude being considered the quantitative 
descriptions obtained by using the scale will have a more 
exact meaning. ‘The measurements yielded by such a scale 
will be more easily handled and interpreted if the differences 
between scale objects are equal. If, in addition, the distance 
of the smallest magnitude from an absolute zero point is 
known, measurements by means of such a scale may become 
as exact as those obtained by using a scale formed by laying 
_off units from a zero point. 
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A quantitative scale of this type could be devised for meas- 
uring the height of boys by selecting a group of boys of ap- 
propriate heights and arranging them in the order of increas- 
ing height. If they were selected so that the differences in 
height between adjacent boys in the scale and the zero point 
were known, the height of any other boy could be described 
quantitatively by determining which scale-boy he most 
closely resembles in height. The boys of the scale could be 
numbered 1, 2, 3, 4,5, etc. The height of a boy not included 
in the scale would be described quantitatively when it is 
stated that his height is equivalent to that of boy No. 11 in 
the scale. 

Standard units. The magnitude of the units which we 
commonly use in the quantitative description of physical 
objects has been fixed by custom, or otherwise, and for this 
reason we speak of them as standard units. For example, 
the magnitude of the yard has been fixed in the United States 
by federal law. It is the distance between two points on a 
metal bar which is kept at Washington. The fixing of stand- 
ard units is not essential to the process of measurement. 
Any magnitude may be arbitrarily defined as a unit by 
anyone. An instrument constructed accurately on the basis 
of such a unit would yield as accurate measurements as 
those which we secure now from instruments based on our 
standard units. Such measures would, however, be under- 
stood only by those who were acquainted with the unit used. 
It is, therefore, obviously desirable that, in the measure- 
ment of physical objects, standard units be used. 

Qualitative characteristics described in quantitative 
terms. Under certain conditions, characteristics of objects 
which are essentially qualitative may be described in quan- 
titative terms. The quality of a sample of handwriting 
is a qualitative characteristic. In order to describe this 
characteristic in quantitative terms it is only necessary to 
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construct a scale consisting of a series of suitable samples of 
handwriting, arranged in order of increasing quality. The 
scale positions of the samples in this array can be defined nu- 
merically by assigning a suitable number to each sample.! 
The quality of any given sample of handwriting could then 
be described in quantitative terms by determining the scale- 
sample which it most nearly resembles in quality, and using 
the numerical name of this scale position as the quantitative 
description of the quality of the sample of handwriting. 

This type of scale is useful for certain purposes; but, 
as we shall show in a later chapter, it is difficult to con- 
struct such a scale, and it is even more difficult to match 
objects which we desire to measure with the correct scale 
objects. It should, however, be recognized that we have 
here a device which, theoretically, makes possible the quan- 
titative description of all characteristics of objects, both 
quantitative and qualitative. 

Description in terms of subjective scales. Another 
method of expressing the description of a qualitative char- 
acteristic in quanitative terms is to estimate the degree to 
which the quality of the characteristic in question ap- 
proaches perfection. For example, the quality of a sam- 
ple of handwriting may be described as 73 per cent of 
perfection. Here the description is in terms of a scale simi- 
lar in certain respects to the one described above. The 
significant difference is that such a scale exists in the mind 
of the person who is making the description and has no ob- 
jective manifestation. Such a scale may be described as 
subjective. Different persons are likely to set up different 
scales. Consequently, they will give different descriptions 
of the same thing. 

Measurement of mental abilities in terms of performance. 


1 The procedure to be followed in constructing this type of scale is de- 
scribed in Chapter VI. 
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We can know of mental ability only as it is manifested in 
some performance. Ability is ability to do. If it cannot 
function, if it cannot produce action, it is not ability in the 
sense in which the term is here used. Thus, it is appropriate 
that we describe quantitatively or measure ability in terms 
of the performance which its functioning produces under 
specified conditions. 

It has been contended that we are not able to measure 
mental ability, that we can measure only performance. This 
distinction, however, is not significant for practical pur- 
poses. As observed in the above paragraph, the meaning 
which we attach to mental ability comes from the per- 
formance which it produces. When the conditions! under 
which the ability functions are specified and controlled, 
i.e., are the same for all children, the measurement of ability 
and the measurement of performance have synonymous 
meanings. If these limitations are kept in mind we may 
speak interchangeably of measuring ability and of measur- 
ing performance. For every performance there is a corre- 
sponding degree of ability. 

The complete description of a performance. A pupil’s 
performance is completely described when three things are 
stated concerning it: 

1. The amount of the performance or the rate at which it 
was given. 

2. Quality or accuracy of the performance. 

3. The character of the exercise in response to which the 
performance was given. In certain cases this description 
may be given the difficulty of the exercise or exercises. 

These three dimensions are implied in the questions: “How 
fast can a pupil work?” “How well can he do the tasks 
set?” “‘How hard tasks can he do?” The relation of a 


1 “Conditions” here include mental states and the procedure which the 
children use in applying their abilities. 
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performance to the ability of which it is the manifestation 
is not defined until the conditions under which the per- 
formance was given are specified. Hence, a statement of 
the conditions under which the ability functioned is an es- 
sential part of its measurement. The three characteristics 
or dimensions of a performance are not always equally 
important, and some measuring instruments are constructed 
so that they yield a separate description of only one or two 
of the characteristics. Other instruments provide for com- 
bining the descriptions of the rate of performance with the 
quality, and some yield scores which are a combination of 
all three dimensions. 

The process of mental measurement. Briefly, the meas- 
urement of an ability consists of securing a quantitative de- 
scription of the performance which that ability produces 
under specified and controlled conditions. Since this ability 
has produced this performance, it is assumed that, under the 
same conditions, it would produce an equivalent perform- 
ance. From this point of view we may think of the meas- 
urement of ability as consisting of securing the quantitative 
description of the performance which it will produce under 
given conditions; but the description of this predicted per- 
formance is, at the same time, a description of a performance 
that has been given. 

Tests and scales. Both of the words, “test” and “scale,” 
have been used in naming our instruments for the measur- 
ing of mental abilities. The measurement of an ability in- 
volves two distinct steps: first, securing from the pupil a 
performance; and, second, describing that performance. If 
we wish to use the words “test” and “scale” in a precise way, 
we should think of a test as being that portion of a measur- 
ing instrument which is used to secure the performance from 
the pupil. The function of a scale is realized in the descrip- 
tion of the performance. The complete measuring instru- 
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ment is, strictly speaking, a test plus a scale. In naming 
our measuring instruments these words have been used with- 
out much discrimination. In so far as there has been any 
discrimination with respect to the precise meanings of the 
words, the term has been used which is characteristic of the 
distinguishing feature of the measuring instrument. For 
example, we have the Courtis Arithmetic Tests, the Kansas 
Silent Reading Tests, and the Thorndike Handwriting Scale. 

Units used in quantitative description of abilities are not 
standard. For describing performances of pupils there are 
no standard units. A large number of units have been used. 
As arule, the makers of our measuring instruments have used 
the units which were most convenient. In describing the rate 
of reading the word or line is generally used as the unit. In 
the operations of arithmetic, the example is the usual unit. 
Words differ in respect to length and to meaning; arithmeti- 
cal examples differ in length and complexity. Hence, these 
units are not standardized in the sense that the yard is a 
standard unit for linear measure. One example may not 
be equal to another example, but a yard is always the same. 
It would be very helpful if a group of standard units could be 
adopted for the measurement of mental abilities; but this has 
not been done. Certain proposals of standard units have 
been made, and these will be described in a later chapter on 
the description of pupils’ performances. 

Assumptions implied in the measurement of mental 
abilities. As indicated above, the procedure of measuring 
mental abilities and the interpretation of the resulting meas- 
ures imply certain assumptions concerning the nature of 
mental abilities and their functioning. Some of these as- 
sumptions have already been suggested, but it will be worth 
while to give definite statement to the significant assump- 
tions in this place. Some of these assumptions depend upon 
the character of the measuring instruments; some of them 
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are introduced by reason of group measurements; others de- 
pend upon the use of common norms for the interpretation 
of all measures yielded by a given measuring instrument. 

1. It is assumed that the performance sustains a constant 
functional relation to the ability which is being measured. ‘This 
assumption is implied in the use of common norms for the in- 
terpretation of all measures yielded by a test. For example, 
if the norm of a given test for the fourth grade is 17, the 
measures of the achievements of all fourth-grade pupils are 
interpreted by comparison with this norm. This is done even 
though the achievements of the pupils are measured by dif- 
ferent examiners at different places and at different times. 
This assumption means that any change in ability from pu- 
pil to pupil produces a proportional change in the perform- 
ances, and that all variations in performances are produced 
by corresponding changes in ability. 

2. It 1s assumed that the performance is the product of the 
functioning of certain mental processes or abilities and no 
others, at least not in a way to disturb the constant functional 
relation. ‘This assumption is implied in the one just men- 
tioned above. In case the performance is to sustain a con- 
stant functional relation to a given ability it is obvious that 
this performance must not be affected by other abilities, 
unless the contributions of these other abilities are synchro- 
nized with the fluctuations produced by the ability being 
measured. Generally speaking, different abilities are suffi- 
ciently independent so that their contributions do not corre- 
late perfectly. This being the case, it follows that the per- 
formance secured must be the product of the functioning of 
certain mental processes or abilities. If other abilities con- 
tribute, their contributions must be slight unless the con- 
stant functional relation is to be grossly disturbed. 

3. Itvs assumed that we can secure the functioning of a given 
ability for the purpose of measurement. 'This assumption is 
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implied in our use of standardized objective tests. When 
a test is given to a group of pupils we assume that we have 
secured the functioning of the same ability in the case of 
each pupil. We, furthermore, assume that each pupil has 
had a reasonable opportunity to demonstrate his ability, un- 
less it is obvious that he has not. In other words, our cus- 
tomary administration of standardized objective tests im- 
plies that we can secure the functioning of a given ability 
whenever we desire it. It is only necessary to present the 
test to the pupils and we shall secure the functioning of the 
ability that we desire to measure. 

4. It is assumed that the testing conditions can be controlled, 
not only through a single testing period, but also on subsequent 
occasions and by different examiners. This assumption is 
fully included in the first one, but it appears to be sufficiently 
important to justify a repetition in thisform. Testing con- 
ditions include the explanation of the test, the timing of it, 
the attitude of the pupils, the effort which they make, the 
lighting of the room, and even the temperature and atmo- 
spheric pressure. In fact, testing conditions include all fac- 
tors which may affect the performances of pupils. It is 
obvious that if these factors are not controlled, so that their 
effect upon the performance is the same for all pupils, even 
when they are tested at different times and by different per- 
sons, the constant functional relationship between the per- 
formance and the ability being measured will be disturbed. 

5. In the measurement of general intelligence it 1s assumed 
that all pupils tested have had equal opportunities to acquire the 
abilities for which the test calls. Although the function of a 
general intelligence test is to yield measures of the native 
or inherited abilities of pupils, it does this indirectly by meas- 
uring certain acquired abilities. Most of the exercises that 
are included in general intelligence tests make demands upon 
school training and upon the general experience of pupils. 
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This assumption should be interpreted in terms of the aver- 
age of all the abilities for which the test calls. It is obvious 
that, in the case of the abilities called for by a particular 
sub-test or a particular exercise, all pupils may not have had 
equal opportunities to acquire them. If the entire test is 
considered as a whole this assumption is approximately in 
agreement with actual conditions for children living in the 
same general environment. 

These assumptions only approximately true. These as- 
sumptions are only approximately true. A perfect con- 
stant functional relation between the ability being measured 
and the performance that is secured is never realized. In 
some cases this relation is disturbed by the effect of other 
abilities, even in cases where it may appear that the supple- 
mentary abilities have been reduced to a minimum. For 
example, it has been demonstrated that the ability to write 
figures may seriously affect the performance of pupils upon 
certain types of arithmetic tests.1_ The functional relation is 
also disturbed because we are not able to exercise complete 
control over testing conditions. This is especially true 
when the test is administered by different persons and at 
different times. 

The fact that these assumptions are not fully realized in- 
troduces errors in the measures we obtain and in their inter- 
pretation. Gradually, our measuring instruments are being 
improved so that these assumptions are more closely approx- 
imated. The important thing is to recognize what assump- 
tions are implied in our procedure, and to know the extent to 
which failure to realize completely these assumptions affects 
the measures which we secure. In a later chapter, under the 
head of validity, we shall consider in detail the procedure 


1See Thorndike, E. L., and Courtis, 8. A. “Correction Formula for 


Educational Testing”’; in Teachers College Record, vol. Xx1, pp. 1-24 (Janu- 
ary, 1920). 
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for ascertaining the extent to which the assumptions are real- 
ized in the case of a particular test. 

A performance which can be observed is required. It is, 
of course, obvious that, for the purposes of measurement, 
it is necessary to secure a performance which can be accu- 
rately observed. Preferably, it should be one which cul- 
minates in a permanent record. Unfortunately, the func- 
tioning of some mental abilities does not normally result in 
such performances. For example, the functioning of the 
ability to read silently does not result in a performance that 
can be observed. In order to secure an observable perform- 
ance in the measurement of this ability it is necessary to in- 
troduce additional abilities. This violates the second assump- 
tion stated above, and thus tends to disturb the constancy 
of the functional relation between the ability and the per- 
formance. When this occurs the resulting measurements 
are likely to be lacking in validity. 

Arguments against the measurement of mental abilities. 
Largely because of the assumptions and difficulties involved, 
some have contended that the measurement of mental abil- 
ities is not possible. In support of their position they have 
asked whether an idea has weight, length, breadth, or 
height. When asked in this form only a negative answer is 
possible. Hence, these critics say an idea cannot be meas- 
ured. The influence of a good teacher, they have claimed, 
cannot be measured. They contend that a mother’s love 
cannot be described in quantitative terms. One fault of 
those who have subscribed to this position has been that 
they were thinking of the measurement of mental abilities 
in terms of the most difficult and indefinite cases with which 
we have to deal. They have, also, failed to recognize in 
their thinking the fact that we define ability in terms of the 
action which it produces. Hence, when we measure a per- 
formance we are indirectly measuring the ability which func- 
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tioned in producing it. Furthermore, they have failed to 
-tealize that since the beginning of the human race, mental 
traits of its members have been measured. We are con- 
stantly measuring our friends and our associates with respect 
to certain traits. A teacher is constantly measuring the 
abilities of his pupils. Such measurements are frequently 
crude, and not always in satisfactory terms. It may be that 
we shall never realize our ideals of validity and accuracy 
in the measurement of certain mental traits. It appears 
that human activity, and particularly the carrying on of the 
work of our schools, require the measurement of various 
human traits. We have always measured them, and it is 
absurd to contend that the measurement of mental abilities 
is impossible. As Professor Thorndike well states, “What- 
ever exists at all exists in some amount.” Measurement is 
simply the description of this amount. 

The limitations of ordinary measurements of mental abil- 
ity. As was pointed out in Chapter I, the measurement 
of mental abilities is not new. The work of our schools re- 
quires that abilities of pupils be measured. However, the 
measurements which we have been making are subject to 
significant limitations. Since educational tests are proposed 
as instruments for securing more satisfactory measurements, 
it is appropriate that we inquire into the limitations of the 
ordinary measurements which teachers are accustomed to 
make. 

Subjectivity of the measuring process is a source of error. 
The process of measurement is said to be subjective when the 
result obtained depends in a large way upon the person mak- 
ing the measurement. This is the case when different per- 
sons measuring the same thing obtain different results. 
When different persons measuring the same thing obtain 
approximately the same result the resulting measures are 
described as objective. Perfect objectivity is not obtained, 
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even in the refined measurement of physical objects. It is 
much less completely realized in the measurement of men- 
tal abilities even when our most refined tests are used. 
Hence, the term “objective,” when applied to a test, must 
be interpreted to mean that the test is relatively objective 
when compared with other instruments which are subjective 
in a high degree. 

If two or more different measurements are obtained for 
the same thing, only one result can possibly be right. It 
may be that all are incorrect. Hence, we shall discuss the 
accuracy of our present school marks in terms of subjectiv- 
ity and objectivity. If we find that different teachers, em- 
ploying similar procedures in measuring the same ability, 
obtain different results, we must conclude that their method 
of measurement is subjective. To say that a certain 
method of measurement is subjective is to say that it tends 
to yield erroneous measures. Some of the measures may not 
involve significant errors, but if the process is highly sub- 
jective most of them will involve errors that cannot be 
neglected. Other causes may contribute to the total error of 
the measures, but proof of their subjectivity is sufficient. 

The subjectivity of examination marks. The written ex- 
amination is one of the most common means of measuring 
the abilities of pupils. It requires that the pupil exercise 
certain abilities in producing a written record of his responses 
to questions and other exercises which the teacher or some 
other school official has prepared. ‘The quantitative de- 
scription of this written record is the pupil’s examination 
“orade” or mark. It may be in terms of percent or of some 
other symbol, such as “A,” “B,” etc. 

Without considering the question of whether a pupil’s 
performance on an examination is a true indication of his 
ability, we may inquire concerning the degree of subjectivity 
of the marks which teachers assign to the examination papers. 
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When the pupil’s performance is a written record we may 
have it described, i.e., measured independently by a number 
of teachers. If their descriptions (“grades’’) agree closely, 
we must conclude that this portion of the process of measure- 
ment by means of the written examination is highly objec- 
tive. On the other hand, if the “grades” assigned to the 
same paper by different teachers differ widely, we must con- 
clude that these “‘grades”’ are highly subjective and, in con- 
sequence, cannot be accurate measures of the abilities of the 
pupils. 

The subjectivity of examination marks has been studied 
by having an examination paper duplicated and then asking 
competent teachers, working independently, to “grade” it. 
Perhaps the best known of these investigations are those by 
Starch and Elliot,! of the University of Wisconsin. They 
studied the marking of examination papers in three school 
subjects, English, geometry, and history. Their method and 
the nature of the facts revealed may be illustrated by an ac~ 
count of their study of the marking of a geometry paper. 

A facsimile reproduction was made of an actual examina- 
tion paper in plane geometry. A copy of this reproduction 
was sent to each of the high schools included in the North 
Central Association of Colleges and Secondary Schools, 
with the request that it be marked on the scale of one hun- 
dred per cent by the teacher of geometry. The teacher was 
asked to mark the paper by the method he was accustomed 
to use. When we consider that the subject-matter of geom- 
etry is quite definite, and that the papers were marked by 
teachers who were thoroughly acquainted with the subject, 
it would seem that we might expect the marks or “grades” 


1 Starch, Daniel, and Elliot, E.C. “Reliability of Grading High-School 
Work in English”’; in School Review, vol. xx, pp. 442-57. “‘Reliability of 
Grading Work in Mathematics”’; in School Review, vol. xxi, pp. 254-59. 
“Reliability of Grading Work in History”; in School Review, vol. xx1, pp. 
676-81. ie 
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placed upon this examination paper to be in close agree- 
ment. However, exactly the opposite was found to be the 
case. Returns were received from 116 teachers. Two 
teachers gave a “grade” above 90, while one “grade” was 
below 30. Twenty were 80 or above, while twenty other 
marks were below 60. Forty-seven teachers assigned a 
mark of passing or above, while sixty-nine teachers decided 
the paper was not worthy of a passing mark. Not only were 
similar results obtained by Starch and Elliot in English and 
in history, but their conclusions have been verified many 
times by other investigators.+ 

In the face of such facts only one conclusion is possible; 
namely, that under ordinary conditions the marks assigned 
to examination papers by teachers are highly subjective. 
Such marks can represent only very crude and very inaccu- 
rate measures of the abilities of pupils. It is not too much 
to say that the mark which a pupil receives on an examina- 
tion paper depends to a large degree upon the teacher who 
“grades” the paper. 

It has been shown also that the same teacher is not con- 
sistent in his own marking. If a teacher “grades” a set of 
papers a second time, the two sets of marks will vary widely.” 

The subjectivity of “final grades.” Teachers compute 
* final grades” in a variety of ways, but they are intended to 
be a quantitative description of the sum total of the achieve- 
ments of pupils within a given subject-matter field. Be- 
cause, under normal conditions, no two teachers have equal 
opportunities for measuring the sum total of the abilities of a 
pupil in the same school subject, we cannot study the sub- 
jectivity of “final grades” by the same method that was 
used in studying the marking of examination papers. Two 


1 See Kelly, F. J. Teachers’ Marks, p. 51, and plowing: for accounts of 


other investigations. 
2 See Starch, Daniel, Educational Measurements, p. 9. } 
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methods which have been used may be illustrated by the: e - 


following investigations. d 

1. Kelly’s investigation. One method has been to com- 
pare “final grades” given to the same pupils by different 
teachers. The limitation of this method is that the two sets 
of measurements cannot be made at the same time. Since 
for a given school subject a pupil is assigned to one teacher, 
it is necessary to compare the “final grades” of two suc- 
cessive semesters or years. The most significant investiga- 
tion of this type is the one made by F. J. Kelly in the public 
schools of Hackensack, New Jersey.? 

In this school system the pupils from four ward schools 
went to a common departmental school (junior high school) 
for the work in the seventh and eighth grades. The pupils 
who in the sixth-A class were taught by four different teach- 
ers were taught in the seventh-B class by the same teacher 
in a single school subject. A comparison of the “final 
grades” given to these pupils in the seventh-B class with 
the ones which they received in the sixth-A class will indi- 
cate the extent to which their “final grades”’ were subjective. 
If the average of the “final grades” given to the pupils in 
one of the ward schools (A) was higher than the average of 
the “final grades” given in another (B), we should expect 
the pupils from the first ward school to receive the higher 
“final grades” when they go to the common departmental 
school, provided the “final grades”? which they received in 
the sixth-A class were accurate measures of their abilities. 
However, if the teacher in school A “graded high,” and the 
teacher in school B “graded low,” i.e., if their “grades” 
were subjective, the average of the “final grades” of the 
pupils from school A may be expected to be below the aver- 
age of those from school B in the seventh-B class. 


1 Kelly, F. J. Teachers’ Marks, Teachers College Contributions to Edu- 
cation, No. 66. - : 
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his latter condition is what Kelly found to exist. He 
tates that “for work which the teacher in school C (one of 
ee the ward schools) would give a mark of ‘G’ in language, pen- 
-manship, or history, the teacher in school D (another ward 
B: school) would give less than a mark of ‘F.’” This means 
_ that the “final grades” given by teachers are subjective, 
Fie., dependent, in part, upon the teacher who gives them. 
Another teacher would give different “grades” for the same 
quality of work. 

2. Johnson’s investigation. Another type of investigation 
of the subjectivity of “final grades” has been made by John- 
son, principal of the University High School of the Univer- 
sity of Chicago. It is based upon the assumption that, 
when accurate measurements are made of any ability of a 
large representative group of pupils, the resulting measures 
_ are distributed, that is, arranged along the scale of measure- 

ment, in a certain definite way. For example, in Fig. 1, 
there are represented graphically four distributions of the 
_ measures of silent-reading ability secured by giving the Kan- 
sas Silent-Reading Tests. The number of measures repre- 
sented in each grade is over five thousand. The base line 
of the curve in each case represents the scale of the test, 0, 1, 
 2,3,4,5,andsoon. At any point of this base line the height 
- of the broken line curve above the base line represents the 
number of pupils having the measure represented. The gen- 
eral shape of these four broken line curves is the same. A 
few pupils received very low measures and a few very high 
ones. The great majority of the measures are grouped near 
the middle where the curve is highest. A curve which, be- 
ginning with the low measures, rises gradually and then falls 
gradually, as do those shown in Fig. 1, is called a “normal 
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1 Johnson, F. W. “A Study of High-School Grades’’; in School Review, 
vol. x1x, pp. 13-24. See, also, Kelly, F. J., Teachers’ Marks, pp. 11, and 
following, for reports of similar investigations. 
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curve.” If the shape of the curve representing the distri- 
bution of a particular set of measures differs materially from 
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the general shape of the curves in Fig. 1, there is reason 
for questioning the accuracy of the measures. 
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PARTMENT. (AFTER JOHNSON.) 


In the University High School, ““F” denotes failure, and 
the four successive ranks above failure are indicated by “D,” 
“C,” “B,” and “A.” For the several departments of the 
school, Johnson tabulated the number of times each mark 
was given during the years 1907-08 and 1908-09. The con- 
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ditions which he found to exist may be illustrated by Fig. 2. 
The upper figure shows the distribution of marks in Eng- 
lish (left) and history (right). It will be noted that in the 
case of English a much larger proportion of low marks (“F” 
and “D’’) were given than in history. For the high marks 
(“A” and “B”’) just the reverse is true. Both curves fail 
to conform closely to the normal curve described above, 
which suggests that the marks are dependent, in part, upon 
the group of teachers who gave them, 1.e., are subjective. 
However, the most significant part of the figure is the 
lower, which represents the distributions of the marks given 
by two teachers in the same department. The distribution 
for teacher A conforms reasonably closely to the normal 
curve, but that for teacher B departs from it in a very con- 
spicuous fashion. If we assume that the two groups of 
pupils were equivalent with respect to their capacity to learn, 
the subjectivity of the marks is apparent. They depend 
upon the teacher who gave them. It is obvious that teacher 
B is accustomed to give “high grades.” In so doing, he has 
furnished evidence that his marks are probably inaccurate. 
Subjectivity of ordinary measures of the abilities of pupils 
a large source of error. Both “examination grades” and 
“final grades” have been shown to be highly subjective; 
that is, it has been shown that different teachers when meas- 
uring the same thing tend to secure different results. This 
means that the “examination grades” or other school marks 
assigned by any teacher are likely to be inaccurate. If they 
are accurate it is largely a matter of chance. There are other 
sources of error, but this is probably one of the largest. 
Other limitations of ordinary measures of the abilities of 
pupils. In addition to their subjectivity, there are a num- 
ber of other limitations of ordinary measures which should 
be noted. Some of these are not a source of error, but rather 
cause misleading interpretations of the measures. Such 
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limitations, however, are significant because measures of 
abilities are of value only as they are correctly interpreted. 

1. Important abilities not always chosen for measurement. 
The field of a school subject includes a large number of abili- 
ties. Careful analyses are necessary in order to determine 
what these abilities are, and which ones are most significant. 
In formulating the questions of ordinary examinations the 
teacher does not, usually, have at hand a statement of the 
important abilities within the field of a school subject, and, 
as a consequence, some of the important abilities are fre- 
quently omitted in the measurement without being recog- 
nized as omitted. Thus, measures of a few abilities are fre- 
quently interpreted as being measures of the entire field of 
abilities. For example, measures of the abilities of pupils 
to do certain types of examples in arithmetic have been in- 
terpreted, when important types of examples were not 
included, as being measures covering the whole field of the 
operations of arithmetic. The making of satisfactory meas- 
urements is dependent upon a careful analysis of the sub- 
ject-matter field in which they are being made. The ordi- 
nary classroom teacher does not have at hand these analyses 
in preparing examinations and in assigning school marks. 

2. No objective norms for interpreting measures. A meas- 
ure has no meaning until it is compared with a norm. The 
size of the measure of a pupil’s ability does not at once tell 
whether a pupil rendered a good performance or not. A 
norm is something which we may use as a basis of com- 
parison. As generally used in connection with the measure- 
ment of abilities of pupils, a norm represents the average 
ability of a certain group of pupils. The comparison of a 
measure with it thus tells one whether the pupil possesses 
average ability, greater than average ability, or less than 
average ability. 

In order to be useful these norms must be stated objec- 
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tively; that is, they must be stated in such a way that all 
teachers will understand them alike. For interpreting or- 
dinary measures there are no objective norms, and a teacher 
must set his own; that is, a teacher uses a norm that is sub- 
jective. This means that it is likely to be different from 
the corresponding norm held by another teacher. Further- 
more, a teacher’s norm probably does not remain constant, 
but changes from time to time. 

In the case of an examination which is marked on the scale 
of 100 per cent, the norm is partly expressed in the passing 
mark, partly in the examination itself, and partly in the plan 
of marking the examination papers. The difficulty of the 
questions and the plan of marking them are intended to be 
such that a pupil who possesses the abilities covered by the 
examination, in a barely satisfactory degree, will! receive the 
passing mark. Assuming that the examination yields a rea- 
sonably accurate measure of the pupil’s abilities, if the list 
of questions is more difficult than the teacher considered it 
_ to be, the pupil will be judged to have a less degree of ability 
than he really has. He will be considered to be below pass- 
ing when he should receive a mark above passing, simply 
because the examination was too difficult. This means that 
the teacher’s norm was too high. The plan of marking the 
examination papers has a similar influence. A severe plan 
of marking will result in lower grades than a more liberal 
one.! 

3. Important dimensions of abilities frequently not measured. 
The complete description of a pupil’s performance requires 
a statement of the rate at which it was produced, its quality, 
and the type or difficulty of the exercises in response to which 


1Jt should be noted that a grade, such as “85 per cent,” or “B,” oF 
“fair,” is a statement of comparison of the measure of the ability of the pu« 
pil with the teacher’s norm. Thus, “grades” should be defined as inter- 
preted measures} i.e., measures compared with the norm. 
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it was given. These characteristics of a performance may be 
called its “dimensions.” In ordinary measurement the rate 
of performance is generally neglected. In many cases, par- 
ticularly in the “tool” subjects of the elementary school, 
the rate isimportant. This condition is illustrated by hand- 
writing, silent reading, and the operations of arithmetic. 
To omit the measurement of the rate in such cases is to 
neglect an important dimension of the pupil’s abilities. In 
some subject-matter fields the rate of work is much less im- 
portant. For example, in solving problems in arithmetic 
the rate of work is less important than the rate at which 
the operations are performed. In such cases failure to meas- 
ure the rate is not a serious omission. 

4. Ordinary measures are not diagnostic. The typical 
school examination covers a variety of topics. Thus, the 
“grade” which a pupil receives is an average or general meas- 
ure. It does not furnish an index of his particular weak- 
nesses. It does not tell in what topics he is strong and in 
what topics he is weak. The ordinary examination is not 
diagnostic. For certain purposes, diagnostic measures are 
required. In such cases the general measures yielded by 
ordinary examinations have a limited usefulness. 


QUESTIONS AND TOPICS FOR INVESTIGATION 


. What is measurement? 

. What do we measure by means of our educational tests? 

. What assumptions are involved in the thesis that we measure the 
abilities of pupils? 

4. What are the characteristics of the abilities of pupils which are sig- 

nificant from the standpoint of their measurement? 

5. What are the arguments of those who contend that we cannot meas- 
ure mental abilities? 

. What are the answers to these contentions? 

. What is a test? What is a scale? 

. What is a standard unit? Are the units used in educational measure- 
ments standardized? 

9. What do we mean by a standardized test? 


09 © 
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10. What are the essential steps in the process of educational measure= 
ment? 

11. What do we need to tell about a pupil’s performance in order to com- 
pletely describe it? 

12. What types of information may we secure about a pupil’s mental 
processes? 

13. What are the limitations of teachers’ marks? 

14. When is a measure subjective? How would you investigate the sub- 
jectivity of teachers’ marks? 
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CHAPTER III 


USES OF EDUCATIONAL MEASUREMENTS IN THE 
WORK OF THE SCHOOL 


General intelligence vs. specific abilities. Before consider- 
ing the uses of educational measurements in the work of the 
school, it will be helpful to make certain distinctions rela- 
tive to the types of measurements which may be made of the 
abilities of pupils. There is, first of all, the measurement of 
general intelligence. For practical school purposes general 
intelligence may be thought of as the measurement of the 
pupil’s general capacity to do the work of the school. Some 
authorities agree with Thorndike that the result of the edu- 
cation provided by the school is the engendering of a num- 
ber of specific or relatively unrelated abilities in the pupil. 
The acquisition of specific abilities implies corresponding 
specific capacities to learn. When the theory of specific 
abilities is accepted, a pupil’s general intelligence is consid- 
ered to be the average of selected samples of his specific 
capacities to learn. Other authorities contend that intelli- 
gence is essentially general, or that it includes some general 
factors. Neither of these concepts of general intelligence 
is incompatible with the working definition stated above. 
Most of our group intelligence tests consist of several differ- 
ent sub-tests. A pupil’s total score is taken as the meas- 
ure of his general intelligence. 

If we accept the theory of specific capacities to learn we 
then need instruments which will yield measures of specific 
intelligence. Such measures are indices of the pupil’s capac- 
ity to learn certain particular things. For example, the 
Rogers Prognostic Tests in mathematics were designed to 
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measure the specific capacity of pupils to learn secondary 
mathematics. Such an instrument may be called a specific 
intelligence test to distinguish it from a general intelligence 
test. Such tests have also been called prognostic tests. 

Types of educational measurements. The majority of 
the tests which have been devised for school use have for 
their function the measurement of achievement, or what 
the pupil has learned. They are called achievement tests. 
Many achievements appear to be specific. In such a field as 
the operations of arithmetic we engender not one general 
achievement, but a large number of achievements which are 
relatively independent of each other. For example, a pupil 
may attain a very high degree of achievement in addition 
of integers without attaining anything like an equivalent 
degree of achievement in division. In fact, each of these 
achievements is general, in a sense, for it has been shown 
that a pupil may attain one degree of achievement in doing 
long-column addition, and exhibit a distinctly different de- 
gree of achievement in doing short-column addition. Since 
ability tends to be specific, we may recognize two types 
of achievement tests; first, those which measure separately 
certain specific achievements; and, second, those which 
yield average or general measures of a number of specific 
achievements within a given field. A test which yields 
separate measures of specific abilities is frequently called a 
diagnostic test, because it points out or diagnoses the specific 
weaknesses of pupils. 

Instruments for measuring the abilities of pupils to en- 
gage in the various types of learning furnish us with a fifth 
type of test. A pupil may possess a high degree of native 
ability, but not be efficient in certain types of learning. For 
example, a pupil may have adequate capacity to learn, but 
be inefficient in memorizing because he goes about this work 
in an inefficient way. There are certain best methods of 
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learning, and, until the pupil has been trained in these 
methods, he is not likely to be efficient as a learner. In- 
struments yielding measures of this kind would tell us the 
degree to which pupils know how to engage in the various 
types of learning which the school requires of them. Rela- 
tively few tests have as yet been devised which have this 
function. Some of our silent-reading tests approximate this 
function for certain types of learning. For example, in the 
Thorndike-McCall Silent-Reading Test, the pupil is given 
a small assignment to read for the purpose of answering cer- 
tain specific questions. This represents one type of learning. 
The Van Wagenen Reading Scales for history and general 
science furnish another illustration of this type of test. 
There appears to be no reason why tests cannot be devised 
for measuring the effectiveness of pupils in other types of 
learning, once the need for them is recognized. 

There is some overlapping between the different types of 
measurements which we have enumerated. A measure of 
achievement is indirectly, also, a measure of intelligence, 
or capacity to learn. A pupil who does not achieve highly is 
usually one who has a relatively low capacity. In general, 
there is a high degree of correspondence between achieve- 
ment and intelligence. This correlation is not perfect by 
any means, because achievement is also influenced by the 
effort which the pupil makes, the instruction which he has 
received, and by other factors. Likewise, there is a positive 
correlation between a pupil’s achievements and the degree of 
effectiveness which he has attained in the methods of learn- 
ing which are required. A high degree of achievement can- 
not be expected from a pupil unless he is efficient as a learner. 
There is, also, a relation between the degree of effectiveness 
in doing certain types of learning and the pupil’s intelligence. 

Plan of analysis to determine the uses of educational 
measurements. The primary purpose of educational meas- 
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urements, whether made by written examinations and other 
crude methods or by standardized objective tests, is to in- 
crease the effectiveness of the work of our schools. The 
construction of standardized objective tests and their use 
can be justified only if we are able to show that the more 
accurate measures of the abilities of pupils which they yield 
will increase the efficiency of our schools. Hence, the first 
step in determining the need for educational measurements 
is to inquire concerning the phases of the work of the school 
which require the measurement of the abilities of pupils. 

School activities requiring measurement. There are a 
number of activities involved in the work of our schools in 
which measurements of the abilities of pupils are required. 
In some of these activities, it is necessary to have informa- 
tion concerning the pupil’s capacity to learn; in others, we 
must know concerning his achievements; and, in still others, 
we need to know concerning the degree of his effectiveness 
in doing certain types of learning. The school activities 
which require measurement may be considered under three 
major heads: 

1. Administrative and supervisory activities. 

2. Instructional activities. | 

3. Research activities. 

The activities which we shall consider under these head- 
ings require the measurement of the abilities of pupils if the 
activities are to be carried on in an efficient manner. Inthe 
absence of standardized objective tests, such as have been 
devised during recent years, the measurements required by 
these activities have been secured by teachers and other 
school officials by employing crude methods, such as the 
traditional written examination. In considering the need 
for standardized objective tests which shall be uniformly 
used by all teachers, it is necessary to bear in mind the pos- 
sibility that the measures obtained by the crude methods 
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may be as effective in some, or even all, of these activities 
as the measures obtained by standardized objective tests. 
Thus, when we show a need for the measurement of abilities 
of pupils, it does not necessarily follow that we have estab» 
lished a need for standardized objective tests. 

Some of these activities are not equally important in all 
grades, and all of them do not require the same type of infor- 
mation. We shall consider these activities in some detail 
with reference to the type of information which they re- 
quire and also with respect to their importance in the differ- 
ent grades. We shall thus be able to indicate, in a general 
way, the types of measurements which are most important 
in the different divisions of our schools. 


1. Administrative and supervisory activities requiring 
measurement 


Promotion and classification of pupils for the purposes of 
instruction. Our public schools are usually organized with 
twelve grades. When a pupil enters school he is assigned to 
the appropriate grade. At the end of the term or year pupils 
who have done the work of their grade satisfactorily are pro- 
moted to the next grade. Occasionally, a pupil receives a 
double promotion, and, in a very few cases, a pupil is sent 
back to a lower grade. In some school systems there are 
occasional cases of promotion during the term. The pro- 
motion of pupils has been and is now based largely upon 
measures of their achievements. Within recent years there 
has been a plea to recognize, also, a pupil’s general intelli- 
gence or capacity to learn. Thus, in this activity there is 
need for both measures of achievement and measures of 
general intelligence. 

Within a school grade pupils may be classified in sections. 
Sections may be formed so that pupils possessing approxi- 
mately the same capacity to learn are brought together for 
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purposes of instruction.! In some school systems special 


classes are formed, such as classes for gifted children, or 
classes for dull and backward children. It has been urged 
that the classification of pupils within a school grade should 
be based primarily upon measures of their capacity to learn 
or their general intelligence. However, measures of achieve- 
ment should not be neglected in this connection. There are 
some pupils who achieve more highly than their general in- 
telligence indicates. This is because of unusual application, 
or other conditions favorable to learning. There are, also, 
pupils of the opposite type who fail to achieve as highly as 
their general intelligence indicates should be expected of 
them. Hence measures of achievement as well as measures 
of intelligence are needed in the classification of pupils. 

Educational and vocational guidance. This activity is 
closely related to the promotion and classification of pupils; 
in fact, below the seventh grade, educational guidance is 
little more than promotion and classification. In the sev- 
enth and eighth grades and, more particularly, in the high 
school the pupil has some choice of the subjects which he 
shall pursue. Thus, there is an opportunity to advise him 
with respect to the work which he shall undertake. Meas- 
urements of general intelligence appear to furnish the most 
important item of information on which to base educational 
guidance. The adviser also needs to have measures of the 
pupil’s achievements as well as other information. Advice 
to pupils concerning the choice of a vocation should be based 
upon information concerning vocations, but frequently meas- 
ures of pupils’ general intelligence and school achievements 
will be valuable. 

Evaluation of school efficiency. It is generally recognized 

1 The author has intentionally avoided any consideration of the desira- 


bility of classifying or segregating pupils on the basis of their general 
intelligence. This question will be considered briefly in a later chapter. — 
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that a high degree of school efficiency cannot be expected 
unless there is an accounting or checking up at intervals. 
The school is, essentially, a manufacturing plant. The pu- 
pils are the raw material; the school buildings, grounds, 
equipment, and textbooks are the plant; the teachers and 
supervisors are the workmen in this educational factory. 
The efficiency of the enterprise depends upon the output, 
that is, the achievements of the pupils. It is also necessary: 
to know the quality of the pupil-material with which the 
school is working. Therefore, in evaluating the efficiency of 
a school, we must measure the general intelligence of the 
pupils as well as their achievements. From one point of 
view, methods of learning constitute achievements; in fact, 
they are among the important achievements of the high 
school. Therefore we need measures of the effectiveness of 
pupils in different types of learning, as well as measures 
of their achievements in school subjects and of their 
general intelligence. The evaluation of the efficiency of a 
school is frequently called a survey. 

The evaluation of the efficiency of the school is a prerequi- 
site for the most effective kind of supervision of instruction. 
Before the supervisor can intelligently prescribe changes 
in the instruction or in the organization of the school, 
he must know concerning the efficiency of the present proce- 
dure. This applies to the entire school system or to any 
unit of it. 

Rating of teachers. The rating of teachers is based, in 
part, on the achievements of their pupils. However, it is 
necessary to take into account the quality of the pupil-ma- 
terial with which a teacher is working before we can form 
any reliable conclusions with reference to a teacher’s effi- 
ciency as an instructor. A teacher who is working with in- 
ferior pupil-material cannot be expected to engender superior 
achievements. Therefore measures of both general intelli- 
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gence and achievements of the pupils must be considered in 
rating teachers. It should be noted that the formal in- 
struction given by a teacher is not the sole source of his con- 
tribution to the child’s education. The teacher’s character, 
philosophy of life, enthusiasm for his work, etc., have a 
profound influence upon many pupils. For this reason it is 
necessary to bear in mind that the rating of teachers should 
not be based solely upon measures of achievement and gen- 
eral intelligence. ; 

Reports to patrons. It is a custom of our schools to send, 
at stated intervals, a report of the pupil’s progress to his 
parents or guardian. The entries on these report cards are, 
for the most part, measures of the pupil’s achievements. It 
is, therefore, obvious that this school activity requires the 
measurement of the achievements of pupils. These meas- 
ures of achievement could be more rationally interpreted if 
they were accompanied by a measure of the pupil’s intelli- 
gence. There are, however, certain reasons why it would 
not be appropriate to include this information on report 
cards. 


2. Instructional activities 


Diagnosis of pupils with reference to achievements. In 
considering this activity of the school it will be helpful to 
recognize two types of instruction. General instruction is 
based upon general principles. It is applied to all members 
of a group alike. The same assignment is given to all; all 
listen to the same explanations. The group is treated as a 
unit. Pupils are different with respect to their past expe- 
riences, and also with respect to their capacities to learn. 
Hence, some pupils will fail to find in this general instruction 
the assistance which they need in their learning; others will 
find this instruction sufficient for their needs. 

Supplementary to this general instruction, there is need 
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for a type of instruction which is based upon the specific 
instructional needs of individual pupils. This type of in- 
struction may be designated as remedial. An obvious pre- 
requisite is the specific measurement of a pupil’s achieve- 
ments. Such measurement is for the purpose of picking out 
the pupils who have failed to achieve and, so far as possible, 
for determining the particular items in which the pupil has 
failed to achieve. Measurement for this purpose is spoken 
of as diagnostic. 

In order that the measures of achievement may be prop- 
erly interpreted, it is necessary to have, also, measures of the 
general intelligence of the pupil. If a pupil has a large ca- 
pacity to learn, his achievements should be correspondingly 
high. On the other hand, if he possesses only a limited ca- 
pacity to learn, we should not expect him to achieve highly. 
In other words, a pupil’s achievements may be expected to 
be commensurate with his capacity to learn. It is only 
when a pupil who possesses the necessary capacity has failed 
to learn that a teacher may expect to find his efforts in the 
direction of remedial instruction rewarded. If a pupil has 
achieved up to his capacity to learn, remedial instruction is 
likely to be fruitless. 

Opportunity for diagnosis varies with the grades. The 
possibility of diagnosing pupils with reference to achieve- 
ments is not the same in all grades. In the first place, pupils 
cannot be diagnosed with reference to their achievements 
until they have had an opportunity to achieve. They must 
have received some instruction on a topic before diagnosis 
is possible.1_ In the elementary school the pupils pursue a 
number of subjects over a period of several years. For ex- 
ample, they study reading in all grades. By repeated drill 


1 This instruction may be incidental. For example, in spelling, a diagnos- 
tic test may be given before the pupils have formally studied a given list of 
words. 
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they are trained to be skillful in the activity of reading. 
Much the same situation exists in spelling, handwriting, 
arithmetic, and, to a less extent, in history and geography. 
In the fields of these subjects there is an abundant opportu- 
nity for diagnosis with respect to achievement. After the 
pupils have received some instruction, a diagnosis with re- 
spect to achievement will be valuable in guiding the teacher 
in the future instruction. 

In the high school, and in some subjects which are taught 
in the upper grades of the elementary school, the situation 
is materially different. When a topic has been studied, 
the pupil does not return to it, except incidentally or in the 
course of review. There are some topics, such as the oper- 
ations of algebra, pronunciation of a foreign language, and 
a few other topics, in which the engendering of skills extends 
over several months or even a longer period. However, 
for the most part, the high-school pupil is engaged in the 
study of topics on which he does not receive continued train- 
ing. Hence the diagnosis of pupils with respect to their 
achievements is impossible until instruction on the topics 
has practically been completed. This condition places 
very obvious limitations upon the usefulness of measure- 
ments of achievement for the purpose of diagnosis in the 
high school. 

Varies also with content of instruction. Another point 
to be considered is that in the elementary school we have 
agreed upon certain minimum essentials as educational 
objectives. Pupils are to be taught to spell the words which 
they will use in written composition. They are to be taught 
the operations of arithmetic. They are to be taught to read, 
and so forth. Although there is not complete agreement 
with reference to the details of these objectives, it is true 
that there is a far greater consensus of opinion with reference 
to objectives in the “tool” subjects of the elementary school 
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than there is in the case of most subjects taught in the high 
school. In history, for example, authorities do not agree upon 
the content of the subject except in the case of a few of the 
most formal items. In fact, it does not appear to be essen- 
tial that the content of such a subject as history should be 
fixed to the extent that the content of handwriting, the oper- 
ations of arithmetic, or the words of spelling should be. It 
appears likely that two teachers of European history might 
be equally efficient in realizing our ultimate educational ob- 
jectives, but vary widely with reference to the emphasis 
which they place upon different topics. Indeed, it is con- 
ceivable that they might not agree completely with reference 
to the topics to be studied. 

The fact that much of the content of the subjects studied 
in the seventh and eighth grades and in the high school is 
not fixed makes it impossible to construct satisfactory in- 
struments for measuring the achievements of pupils. A test 
which would measure the outcomes recognized by some 
teachers as minimum essentials would be criticized by 
other teachers as failing to measure other outcomes which 
they considered essential. Until there is a greater degree 
of agreement concerning the minimum essentials of the sub- 
jects taught in these grades, it will not be possible to con- 
struct standardized objective tests which can be recom- 
mended for general use in diagnosing pupils with respect to 
their achievements. This condition does not remove the 
need for diagnosis, but it should be made by instruments and 
methods which are adapted to the instruction which the 
pupils have received. 

Finally, many of the outcomes of instruction in the 
seventh and eighth grades and in the high school are of 
such a nature that the problem of measurement is distinctly 
more difficult than for the more formal outcomes in the 
elementary school. We have not yet been able to devise 
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instruments for measuring these outcomes of instruction 
which yield results that are as satisfactory as those obtained 
for the more formal subjects. This is particularly true in 
the case of diagnostic tests. 

Thus, it appears that the diagnosis of pupils with refer- 
ence to achievement in the field of content subjects by means 
of uniform instruments is limited with respect to opportunity 
and usefulness. Diagnosis is needed, but until we have more 
completely agreed upon the specific educational objectives to 
be attained in content subjects and consequently attempt 
to produce a uniform product, each teacher must diagnose 
his pupils, for the most part, by means of methods and in- 
struments which may be adapted to the outcomes he is en- 
deavoring to produce. Furthermore, instruction in these 
subjects is at present so organized that teachers would be 
able to make only a very limited use of diagnostic informa- 
tion, even if it were available. Hence, in diagnosing pupils 
with reference to achievement, the demand for measure- 
ment by means of uniform standardized tests is confined 
largely to the “tool” subjects of the elementary school. At 
the present time this activity makes little demand for such 
instruments in the high school. 

Diagnosis of pupils with respect to study procedures. 
The work of the school requires of the pupil a number of 
different types of study. For example, in handwriting, in 
the operations of arithmetic, in reading, and in similar sub- 
jects, he is asked to acquire skill so that he can do exercises 
with a high degree of fluency. In other subjects he is asked 
to solve problems and to acquire abstract and general mean- 
ings. In still others the pupil is expected to acquire ideals. 
The learning activity required of pupils is not the same for 
the different objectives. Certain study procedures are re- 
quired for memorizing; others for acquiring ideals. Before 
pupils can be efficient in their study, they must have ac- 
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quired a good study technique. They must know how to go 
about the study of an assignment. 

One of the objectives of the school is to train the pupil in 
methods of learning. In fact, it may be urged that this is 
the most important objective of the secondary school. The 
term “preparatory,” which for a long time was used to de- 
scribe the school which the pupil attended before entering 
college, definitely indicates this function. If a pupil has 
acquired a good technique of learning, he possesses a very 
important type of preparation, not only for college, but also 
for participation in the activities of adult life. 

Need for diagnostic study tests. It is obvious that in effi- 
cient instruction there is need for the diagnosis of pupils with 
respect to their effectiveness in the types of learning which 
the school requires of them. Pupils must be efficient as 
learners before they can be expected to attain the objectives 
of the school. The fact that we have just made a beginning 
in devising instruments for this type of measurement should 
not be interpreted to mean that the demand for this type of 
measurement is of minor importance. 

In the elementary school the pupil’s learning is largely 
that required for the acquisition of skills. This procedure 
is relatively simple. It is relatively easy to ascertain by 
observation whether or not a pupil is following a good pro- 
cedure. Hence, there is little need for this type of measur- 
ing instrument in the elementary school below the seventh 
grade. In the high school the pupil is required to engage 
in more complex types of learning. Furthermore, it is not 
easy to determine by observation whether or not a pupil 
is engaging in the type of learning which will lead to the so- 
lution of a problem or the acquiring of an ideal. There is, 
therefore, a greater need than in the elementary school for 
instruments which teachers may use for diagnosing pupils 
with respect to their study habits. 
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3. Research activities 


Determination of procedure through measurement of 
outcomes. The scientific attitude applied to education 
requires that methods and devices of teaching, courses of 
study, textbooks, and other items of educational procedure 
be chosen on the basis of their effectiveness in achieving the 
objectives of education. Therefore, any determination of 
what procedure is best in education requires the measure- 
ment of the outcomes secured by means of the procedures 
which are being studied. For example, if one wishes to de- 
termine the merits of a teaching device, one must apply this 
device to a group of pupils and measure the outcomes pro- 
duced by it. Only in this way may we know its worth. Like- 
wise, we can learn the worth of other items of procedure, 
including types of school organization, only by measuring 
the achievements of pupils who have been subjected to the 
procedure. 

In any research activity it is necessary to take into ac- 
count the quality of the pupil-material as well as the charac- 
ter of the achievement secured. Therefore, we have here a 
demand for both types of measurements. Research is im- 
portant both in the elementary school and in the high school. 
The opportunities for research are, however, somewhat 
greater in the former, because we have more satisfactory 
measuring instruments for this field. 


‘General summary. The measurement of general intelli- 
gence is required in practically all activities in which there is 
need for any measure of the abilities of pupils. This informa- 
tion is always needed in the interpretation of achievement. 
It is, likewise, needed for the interpretation of measures 
of the effectiveness of pupils in the different types of learn- 
ing. The measurement of general intelligence is needed in all 
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divisions of the school. It is difficult, if not impossible, to 
say that the need is greater in certain grades than in others. 
If the policy of classifying and promoting pupils largely on 
the basis of their general intelligence is followed, the need 
for measures of this trait is naturally greater. 

The demands of administrative and supervisory activities 
for the measurement of achievements do not vary with the 
different grades except as these activities vary in importance. 
For the reasons which we have pointed out, the usefulness of 
diagnostic measurement with respect to achievement is lim- 
ited, for the most part, to the elementary school. On the 
other hand, the measurement of the effectiveness of pupils 
in different types of learning has a greater importance in the 
high school than in the elementary school. 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1. What types of information may we secure about the mental processes 
of pupils? 
2. Distinguish between general intelligence and achievement. 
8. Distinguish between general intelligence and specific intelligence. 
4. Distinguish between diagnostic measurement and general measure- 
ment. 
. What is prognostic measurement? 
. What activities of the school require the measurement of abilities of 
pupils? 
7. What types of measurement are required by each activity? 
8. What is the relative importance of measurement in these activities in 
the different divisions of the school? 
9. Give illustrations of the use of measurement in these school activities. 
10. Why do we have a much larger number of tests for the elementary- 
school field than for the high-school field? 
11. What are the most important types of tests for use in the elementary 
school? What are the most important types for use in the high school? 
12. Describe remedial instruction. Give illustrations of remedial instruc- 
tion. 
13. What kinds of measurement should be made in a school survey? 
14. How are educational measurements related to the work of the super- 
visor? 
15. State the types of measurement most useful in the elementary and 
in the secondary school fields, and distinguish between them. 


D> or 


54 THEORY OF EDUCATIONAL MEASUREMENTS 


SELECTED REFERENCES 


Buss, D. C. “The Application of Standard Measurements to School 
Administration”; in Fifteenth Yearbook of the National Society for the Study 
of Education, part 1, 1918. 

Branson, Ernest P. ‘An Experiment in Arranging High-School Sec- 
tions on the Basis of General Ability ’’; in Journal of Educational Research, 
8: 53-55 (January, 1921). 

Courtis, S. A. “Objective Standards as a Means of Controlling In- 
struction and Economizing Time”’; in School and Society, vol. 1, pp. 433— 
36 (March 27, 1915). 

Courts, S. A. “Supervisory Control by Means of Objective Standards”; 
in Proceedings, Second Indiana Conference on Educational Measuremenis, 
1915, pp. 37-68. 

Courtis, S. A. ‘Educational Diagnosis”; in Proceedings, Second Indi- 
ana Conference on Educational Measurements, 1915, pp. 135-71; also in Educa 
tional Administration and Supervision, vol. 1, pp. 89-116 (February, 1915). 

Courtis, S.A. ‘‘Courtis Tests in Arithmetic: Value to Superintendents 
and Teachers”; in Fifteenth Yearbook of the National Society for the Study 
of Education, part 1, 1916, pp. 91-106. 

Dickson, Viratt E., and Norton, Jonn A. “ The Otis Group Intelli- 
gence Scale Applied to the Elementary-School Graduating Classes of Oak- 
land, California”; in Journal of Educational Research, 3: 106-15 (February, 
1921). 

Dickson, Virett E. ‘The Use of Group Mental Tests in the Guidance 
of Eighth-Grade and High-School Pupils”; in Journal of Educational Re- 
search, 2: 601-10 (October, 1920). 

Haacrrty, M. E. “Some Uses of Educational Measurements’’; in 
School and Society, vol. tv, pp. 762-71 (November 18, 1916). 

Haccerty, M. E. ‘“ Measurement and Diagnosis as Aids to Super- 
vision”; in School and Society, vol. vi, pp. 271-85 (September 8, 1917). 

Kirxratricx, E. A. “Measurements, Standards, and Teaching”; in 
School and Society, vol. 1, pp. 278-81 (February 20, 1915). 

Mapssn, I. N. “‘Group Intelligence Testsas a Means of Prognosis in 
High School”; in Journal of Educational Research, 3: 43-52 (January, 1921). 

Mutter, W.S. “The Administrative Use of Intelligence Tests in the High 
School”’; in T'wenty-first Yearbook of the National Society for the Study of Ed- 
ucation, part 11. Bloomington, Illinois; Public School Publishing Com- 
pany, 1922, pp. 189-222. 

Proctor, W. M. “The Use of Psychological Tests in the Educational 
Guidance of High-School Pupils”; in Journal of Educational Research, 1: 
369-81 (May, 1920). 

Proctor, W. M. “The Use of Psychological Tests in the Vocational 
Guidance of High-School Pupils”; in Journal of Educational Research, 2: 
533-46 (September, 1920). 


USES OF EDUCATIONAL MEASUREMENTS 55 


Starca, D. “Standard Tests as Aids in the Classification and Pro- 
Motion of Pupils”’; in Fifteenth Yearbook of the National Society for the Study 
of Education, part 1, 1916, pp. 143-48. 

Srrarer, G.D. ‘‘The Use of Tests and Scales of Measurements in the 
Administration of Schools’; in Proceedings of the National Education 
Association, 1915, pp. 579-82. 

Tuetsen, W. W. “The Relative Progress of VII-B Groups Sectioned on 
the Basis of Ability’; in Journal of Educational Research, 5: 295-305 
(April, 1922). 

Trasur, M.R. “The Use of Intelligence Tests in Junior High School”; 
in Twenty-first Yearbook of the National Society for the Study of Education, 
part m. Bloomington, Illinois: Public School Publishing Company, 1922, 
pp. 169-88. 

Warrriz,G.M. “The Use of Mental Tests in the School”; in Fifteenth 
Yearbook of the National Society for the Study of Education, part 1, 1916, pp. 
149-59. 


CHAPTER IV 
THE CONSTRUCTION OF EDUCATIONAL TESTS 


General structure of educational tests. Practically all 
educational tests either consist of structural divisions called 
exercises, or, for the purpose of description, certain divisions 
are made. We may, therefore, think of the exercise as a 
structural unit of the test. The different types of exercises 
and their arrangement produce several types of tests. The 
test also includes the specifications of the conditions under 
which the exercises are presented to the pupils. 

Types of exercises. Exercises for testing purposes vary 
in a number of respects. Some of them call for the func- 
tioning of the ability being measured under approximately 
normal conditions; others are highly artificial. Some exer- 
cises are easy; other exercises are very difficult. Some re- 
quire only a few seconds for the pupil to answer; others re- 
quire several minutes. Some require much writing; others 
may be answered by making a single mark. 

In the operations of arithmetic it is a simple matter to 
construct an exercise which will call for the functioning of a 
given ability. Furthermore, the exercise constructed will 
approximate the normal conditions under which this ability 
functions. In most cases, if he so desires, the test-maker can 
easily construct an unlimited number of exercises of this 
same type which will call for the functioning of the same 
ability, but which are different in the sense that the answers 
to the examples will be different. This is the simplest 
type of exercise construction. It is illustrated by the 
exercises of such tests as Monroe’s Diagnostic Tests in 
Arithmetic, or the Cleveland Survey Tests in Arithmetic. 
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Spelling ability normally functions in the spelling of words 
as they occur in sentences, and when the attention is focused 
primarily upon formulation of the ideas which the sentences 
express. Since it is desirable for testing purposes that all 
pupils spell the same words, it is impossible to construct 
a set of exercises which will require the spelling of a list 
of words under completely normal conditions. A spelling 
test consisting of a list of words to be pronounced separately 
to the pupils calls for their spelling out of any of context, 
and the attention of the pupils will be focused primarily upon 
the spelling. If the test words are embedded in sentences, 
normal spelling conditions are more nearly, but probably not 
closely, realized. Hence, in the field of spelling, test exer- 
cises must be artificial in the sense that they do not closely 
approximate the conditions under which spelling ability 
normally functions. 

Devising tests to give an objective record. Insome school 
subjects the normal functioning of the ability does not pro- 
duce an objective record. In such cases it is necessary to de- 
vise an artificial exercise which requires the functioning of the 
ability to be measured and which will give an objective rec- 
ord. For example, the results of the process of silent read- 
ing under normal conditions are not observable. Certain re- 
sults, such as posture, facial expression, and the functioning 
of some of the vital organs may be observed in certain cases, 
but such performances are not satisfactory for testing pur- 
poses. In testing silent-reading ability it therefore becomes 
necessary to devise artificial exercises which require not 
only silent reading, but also other abilities whose function- 
ing will yield a performance that may be observed and con- 
veniently described. One such type of silent-reading exer- 
cise is to have the pupil read a paragraph and then answer a 
question on it. Another type is to have the pupil reproduce 
what he has read. In both of these types the pupil is asked 
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to exercise abilities other than those involved in silent read- 
ing. 

The examination of available educational tests reveals a 
large variety of types of exercises. In school subjects where 
the usual classroom exercises are not suitable for testing pur- 
poses, test-makers have exhibited much ingenuity in invent- 
ing types of exercises which are suitable for testing purposes 
and which at the same time satisfy other requirements of 
test construction. A number of typical exercises from rep- 
resentative silent-reading tests are reproduced below to illus- 
trate what is meant. 


MONROE’S STANDARDIZED SILENT READING TESTS 
REVISED 
At every turn the maples burn, 
The quail is whistling free. 
The partridge whirrs and the frosted burrs 
Are dropping for you and me. 


What season of the year does the stanza tell about? Draw a 
line under the one you think. 


spring summer autumn winter 
In front the purple mountains were rising up, @ distant wall. 
Cool snow gleamed upon the summits. Our horses suffered bit- 


terly for water. Five hours we had ridden through all that arid 
waste without a pause. 


What kind of a country had these people been riding through? 
mountainous swampy desert forest valley 


THORNDIKE-McCALL READING SCALE 
Read. this and then write the answers. Read it again if you need to 


According to the Kansas City Star, the wheat farmers of Kan- 
sas are too prosperous to trouble themselves about careful har- 
vesting. They do not cut the fields clean. A gleaner 80 years old, 
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after the wheat harvest in Pawnee County last year, went over the 
wherdt-fields with a wagon, a rake, a brush, and a shovel and swept 
up the wheat left on the ground by the threshers, He gathered 
nine hundred bushels in forty days and sold it at a dollar a bushel. 


Do you think most of the wheat which the farmers grew was 


left on the ground by the threshers ?,...... Se a ee ‘ 
Why are the flelds not cut clean ?.....ccccccccsccevncesucs 
” Where did the gleaner spend the Bei GUYS Pas snes thiantin 


HAGGERTY READING EXAMINATION, SIGMA 8 


The champions were therefore prohibited to thrust with the 
aword, and were confined to striking. A knight, it was announced, 
might use a mace or battle-axe at pleasure, but the dagger was a 
prohibited weapon. A knight unhorsed might renew the fight on 
foot with any other on the opposite side in the same predicament; 
but mounted horsemen were in that case forbidden to assail him. 
When any knight could force his antagonist to the extremity of the 
lists, so as to touch the palisade with his person or arms, such op- 
ponent was obliged to yield himself vanquished, and his armor and 
horse were placed at the disposal of the conqueror. A knight thus 
overcome was not permitted to take further share in the combat. 
If any combatant was struck down, and unable to recover his feet, 
his squire or page might enter the lists and drag his master out of 
the press; but in that case the knight was adjudged vanquished, 
and his arms and horse declared forfeited. 


1. Underline the word which names the weapon that could not be 
used: 
sword | 
mace 
dagger 
battle-axe 
@. Check the one of these statements which is false: 
a. — A knight could fight on foot. 
b. — One knight could not injure another knight. 
ec. — Mounted horsemen could fight only mounted horsemen. 
8. Check the false statements; 
a. — A knight could be vanquished without being killed. 
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b. — A knight’s page could fight. 
ce. — A vanquished knight retained his horse. 
4, Check the true statements: 
a. — Champions were prohibited to use the sword. 
b. — An unhorsed knight could renew the fight. 
c. — An opponent was vanquished if his arms touched the 
palisade. 
d. — A knight dragged from the lists by his page was beaten. 


VAN WAGENEN READING SCALES, GENERAL HISTORY 
. SCALE A 


Read the paragraph carefully. Then read the statements below 
at and puta check mark (V) on the dotted line in front of each 
statement which contains an idea that is in the paragraph or can be 
derived from it. The paragraph and the statements may be re-read 
as often as it is necessary. 


The factory system, introduced in America at the beginning of 
the century, was well developed by the time of Jackson’s presi- 
dency. The motive power to run the machinery was obtained 
almost entirely from the swift currents. To accommodate them- 
selves to the new principles of industrial organization, the working 
classes found it necessary to lay aside the small domestic manu- 
facturing which they had previously carried on in their scattered 
and isolated homes, and to gather themselves under a common roof, 
for common effort. A new system of labor was evolved, for the 
concentration of artisans meant the appearance in the community 
of a unique social class, possessed of its own special needs, which 
were different from those of any other class. The factory owners 
soon realized the exigencies of the new situation, and erected both 
boarding-houses for the accommodation and protection of the 
young women in their employ and separate tenements for the 
married employees and their families, while at the same time they 
made generous contributions for the support of the schools and 
churches in the community. The employees were frequently paid, 
in whole or in part, in “orders” on the company store, where com- 
modities for their use were on sale. A time-table of the Lowell 
mills for the year 1852 shows that in the month of June of that year 
the first bell of the factory awakened the operatives at four-thirty 
in the morning, a second bell came at four-fifty, and the day’s work 
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began by the third bell, early enough to allow of a work period of an 

hour or so before breakfast. 

ie eR 1. Previous to the introduction of the factory system 
manufacturing was done in the homes. 

Rr sheties 2. Previous to the introduction of the factory system the 
American homes were scattered rather than in large 

- groups. 

Aa Eee Te 3. Jackson was President of the United States at the time 
the factory system was first introduced. 

Risre tens tics 4. The first kind of power used in American factories was 
obtained from the rivers. 

Sees sa 5. The factory system was introduced into America 
about 1700. 


Definition of difficulty. Difficulty has been defined as that 
characteristic of an exercise which, when present in a large 
degree, causes a large per cent of incorrect responses, and, 
when present in a small degree, is accompanied by a small 
per cent of errors. In other words, the degree of difficulty of 
an exercise is determined by the per cent of correct responses 
obtained when it is given to a large number of pupils.’ If 
certain assumptions are made concerning the distribution of 
the ability of the pupils to whom an exercise is given and the 
zero point is located, the degree of difficulty of the exercise 
can be expressed in terms of a unit of variability of this dis- 
tribution of ability. This unit is the difference in difficulty 
between two exercises which are answered correctly by cer- 
tain per cents of a given group of pupils. The median devi- 
ation (P.E.) is frequently used as a unit. It is defined as the 
difference in difficulty between an exercise which is answered 
correctly by 50 per cent of the pupils, and an exercise which 
is answered correctly by 25 per cent of the same pupils. The 
standard deviation (S.D. or @) is also used asa unit. It isthe 
difference in difficulty between an exercise answered cor- 


1 The measure of accuracy in terms of per cent is with reference to stand- 
ards which may or may not be the same. See page 110. 
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rectly by 50 per cent of the pupils and an exercise answered 
correctly by 15.87 per cent of the same pupils. Thus, we de- 
scribe the difficulty of exercises as being 2.7 P.E., 6.3 P.E., 
5.20,etc. The method of determining the difficulty of an 
exercise will be described on page 95. 

Types of tests produced by difficulty of exercises. When 
the exercises of a test are equal with respect to difficulty, the 
performance is uniform and the test may be called a uniform 
test. The Courtis Standard Research Tests, Series B, is an 
illustration of this type of test. 

If the exercises of a test vary in difficulty, and are ar- 
ranged in ascending order of difficulty, we have a scaled 
test. Usually, in a test of this type an effort is made to have 
a wide range of difficulty represented, and to have the differ- 
ences between successive exercises equal. Hence, it is neces- 
sary to consider the difficulty when selecting exercises for 
such atest. The Woody Arithmetic Exercises are typical of 
this kind of measuring instrument. Instead of a single exer- 
cise on each level of difficulty, we may have a group of ex- 
ercises which constitute a uniform test. In fact, this struc- 
ture is implied when a scaled test is constructed for individual 
testing. The Thorndike Visual Vocabulary Scales illustrate 
this type of structure. 

If, instead of the arrangement in ascending order of diffi- 
culty, the exercises are arranged without regard to difficulty, 
we have an irregular test. In tests of this type, extremely 
easy and extremely difficult exercises are usually not in- 
cluded, and the exercises of the test are selected on some 
basis other than difficulty. If the irregularities are relatively 
small, the test may be treated as being uniform without in- 
troducing serious error. This type of test may be illustrated 
by Charters’ Diagnostic Language and Grammar Test, and 
by Monroe’s Standardized Reasoning Tests in Arithmetic. 

A cycle test is secured when different types of exercises 
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recur in the test at regular intervals. The exercises of a 
given type are assumed to be equal in difficulty or approxi- 
mately so. The Illinois Standardized Algebra Tests repre- 
sent this principle of test construction. 

The name, spiral test, has been applied to a modification 
of the scaled test. A spiral test consists of a number of sub- 
tests. The exercises within each sub-test are uniform with 
respect to difficulty or, at most, moderately irregular. The 
sub-tests differ from each other in difficulty, but they are 
usually selected on the basis of content rather than difficulty. 
Generally, the scores of the sub-tests are kept separate. 
Hence, most of our spiral tests incorporate the characteris- 
tics of a scale only in a very crude form. They are essen- 
tially nothing more than a battery of uniform tests. The 
term, spiral test, was first applied to the Cleveland Survey 
Tests. « 

Power tests and rate tests. The name, power test, has 
been applied to scaled tests which yield measures of ability 
in terms of the difficulty of the exercise or group of exercises 
done with a specified degree of excellence. For this type of 
test pupils are allowed to continue their work until they are 
unable to do any more exercises correctly. The Thorndike 
Visual Vocabulary Scale and the Van Wagenen Reading 
Scales are examples of power tests. With only a slight mod- 
ification of meaning, the name, power test, could be applied 
to other types of tests when unlimited time is allowed. The 
measure of ability would then be in terms of the quality or 
accuracy of the performance. 

A rate test is defined as one which yields a measure of the 
pupil’s rate of work. Frequently a measure of the quality 
of the work is also secured. 'To measure the rate of work it 
is necessary either to set a time limit, such that practically no 
pupil can complete all of the exercises, or to time each pupil 
separately. In a few rate tests the pupils are allowed to 
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complete the test after they have indicated the place reached 
at the end of the time interval allowed. Rate tests are il- 
lustrated by the Courtis Standard Research Tests, Series B. 
A time limit has been applied to all types of tests, but the 
measure of rate which is obtained has little meaning ex- 
cept for uniform tests, or irregular and cycle tests which 
approximate uniformity. 

Requirements governing the construction of educational 
tests. It is highly desirable, for practical reasons, that the 
administration of a test be simple and that it not require a 
large expenditure of time, but the prime requisite is that it 
yield scores which are true indices of the ability or group of 
abilities which it is designed to measure. It has been urged 
that when this condition is met the details of the structure of 
the test and the content of the exercises are matters of minor 
importance. Unfortunately, it is impossible, or at least ex- 
tremely difficult, to prove conclusively by direct methods 
that a test does or does not measure that which it is intended 
to measure. In some cases partial evidence of the validity 
of a test can conveniently be secured; but for practically all 
tests there is need for supporting evidence, such as is fur- 
nished by an analysis of the abilities measured, the content 
of the exercises, and the structure of the test. 

The production of inferior measuring instruments will be 
reduced to a minimum by recognizing certain requirements 
governing the construction of educational tests. Some of 
these requirements are implied in the assumptions noted in 
Chapter II. Others refer to the use of the information which 
the tests yield. In the list of requirements! given below 
there is some overlapping, but it is thought that these state- 


1 These requirements are not stated as criteria for evaluating a test after 
it has been constructed. They are “guides” to be followed in the construc- 
tion of educational tests. The evaluation of a test is discussed in Chap- 
ter IX. 
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ments will prove helpful “guides” to one who is attempting 
to construct an educational test. An elaboration and dis- 
cussion of these statements is given in the following pages. 

1. The performance should maintain a constant functional 
relation to the ability or group of abilities being measured. 
This is acomprehensive requirement. Several of the others 
stated here may be considered subordinate to it. This re- 
quirement will be recognized as implied in the first as- 
sumption stated in Chapter II. 

2. The test should provide adequate opportunity for all 
pupils to demonstrate their abilities in the field defined by 
its function. 

3. In tests designed to measure school achievement, the 
performance should be consistent with recognized educa- 
tional objectives. 

4. The performance should be one that is conveniently 
secured, recorded, and described. 

5. The test should provide for controlling the conditions 
under which the performance is given. This control should 
be such that the same testing conditions will be approximated 
by different examiners in different places and at different 
times. 

6. The test should make possible the description of a 
pupil’s performance in terms of the dimensions or charac- 
teristics that are significant. 

7. In constructing a scaled test, the exercises or groups 
of exercises should be equally spaced upon the scale of diffi- 
culty. 

It is not always possible to meet completely these require- 
ments. Frequently, it is necessary to effect a compromise. 
However, they should be thought of as “guides”’ in the con- 
struction of educational tests. In general, the more com- 
pletely these requirements are met the better the resulting 
test will be. 
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Nature of ability. Although the available analyses of 
ability are unsatisfactory, our best information indicates 
that there is not one ability in a subject-matter field, but 
many abilities which are relatively separate and distinct. 
At least, they may be so considered for practical purposes. 
They may involve a general factor, as some authorities con- 
tend, but it appears that they include at least some elements 
which are specific. For example, there may be general fac- 
tors included in the abilities required for the doing of differ- 
ent types of examples in the field of arithmetic. It, however, 
appears that the ability to do one type of example is not 
necessarily accompanied by equivalent ability to do another 
type of example. Therefore, there are at least some ele- 
ments of abilities which are not general, but specific. . 

The development of ability. Ability develops in two 
ways. Forexample, the child learns to read simple material. 
Practice tends to make him more fluent in this process. He 
is able to increase his rate of reading and, to some extent, 
the degree of his comprehension. This is one way in which 
silent-reading ability develops. The pupil also increases 
his ability to read by becoming able to read more and more 
difficult material. Thus, we need to recognize two types of 
growth, one in the direction of fluency of functioning of cer- 
tain abilities, and the other in the direction of the develop- 
ment of abilities to do more and more difficult things of the 
same general type. Educational tests are designed to meas- 
ure the growth or development of abilities. Some tests have 
for their purpose the measurement of a pupil’s fluency in a 
given field; others are used to determine the range of a pupil’s 
ability. Power tests belong to the latter class. 

Relation of performance to ability. The performance 
which a pupil gives in response to a test depends upon a num- 
ber of factors. The abilities which the exercises of the test 
call for may be limited to the ones which we are attempting 
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to measure. In this case we have direct measurement. 
Most educational measurements are, however, semi-direct 
or indirect, and the abilities being measured are combined 
with others required by the test. For example, in the meas- 
urement of silent-reading ability, it is necessary to require 
the pupil to reproduce what he has read, answer questions, 
or give some other performance that can be observed. The 
functioning of abilities other than those involved in the 
process of silent reading is required to produce such per- 
formances. It has recently been shown that a performance 
consisting of the writing of the sums of the fundamental 
addition combinations depends upon the ability to write 
figures, as well as upon a knowledge of the number combina- 
tions.1_ It was found that some pupils were limited in the 
scores which they were able to make by their slow rate of 
writing the answers to the number combinations. Had they 
been able to write more rapidly their scores would have been 
materially increased. On the other hand, there were pupils 
who were able to write more rapidly than they could give 
the number combinations. Since the rate of writing figures 
is variable from pupil to pupil, the performance is a func- 
tion of the rate of writing figures as well as of the ability to 
recall the fundamental addition combination.? 

The performance is also affected by a number of other fac- 
tors, such as the effort which a pupil makes, his physical con- 
dition, his emotional status, the time of day, the form in 
which the test is presented, the manner of the examiner, the 
explanations and directions given to the pupil, acquaintance 


1 Courtis, S. A., and Thorndike, E. L. ‘Correction Formule for Addi- 
tion Tests”; in Teachers College Record, vol. xxt, pp. 1-24 (January, 1920). 

2 The investigators propose a correction formula by means of which the 
performance which a pupil would make if he were not handicapped by 
the writing of the answers can be calculated. Such a correction formula is 
useful for experimental purposes, but it introduces a procedure which 
is too complex for general use. 
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with testing procedure and the particular type of exercises 
used, specific coaching for the test, the time allowed, and 
other more subtle factors. The performance is, therefore, a 
function of a number of variables. The functional depend- 
ence of the performance can be represented in mathematical 
terms by the following equation: 


P= FlOj, Casas co Gnu Ses eat ae 


In this equation, the a’s represent the abilities to be meas- 
ured and the 2’s all of the other factors which affect the 
performance. 


1. Requirements for constant functional relationship 


How to secure this. A constant functional relationship 
between the performance and the abilities being measured 
means that the same relation exists for all pupils, even when 
tested by different examiners and at different times and at 
different places. The necessity for this constancy of rela- 
tionship is due to the use of the same set of norms for inter- 
preting the scores of all pupils. The first step in securing 
a constancy of functional relation is to specify the abilities 
to be measured. Our purpose may be to secure a general 
or average measure of the abilities within the entire field of 
a school subject, or even within a large section of this field. 
On the other hand, we may desire a measure of the abilities 
within a very restricted field, such as single-column addition 
of integers, seven figures to the column. Whatever field of 
abilities is chosen, the exercises should be limited to that 
field and should be representative of it. In the case of gen- 
eral measurement, this requires that they form a random 
sample of it. The use of a variety of types of exercises 
within a single test will result in the score being an unana- 
lyzed total, unless the test is sufficiently long and the types 
are distributed in a random way. In such a case, the score 
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may be looked upon as an average or general measure of the 
abilities in the field represented by the exercises of the test. 

The abilities introduced for testing purposes, of course, 
are variable from pupil to pupil. The other z-factors also 
exhibit variable tendencies unless proper precautions are 
taken. In order that the performance shall approximate a 
constant functional relation to the abilities measured, it is 
necessary that the effect of other abilities be eliminated or 
reduced to a minimum, and that the other z-factors be con- 
trolled so that there will be a minimum of variation from 
pupil to pupil. This ideal can only be approximated, but 
the test-maker should recognize the importance of the con- 
trol or elimination of all variable factors except the abilities 
being measured. 

Excluding other abilities than those it is desired to test. 
The exercises should be constructed so that abilities other 
than those being measured will be involved to the smallest ex- 
tent consistent with other requirements of test construction. 
For example, exercises should be constructed so that a mini- 
mum of writing is required. When possible, it should be 
reduced to checking or marking certain words or statements, 
The Burgess Picture Supplement Scale for Measuring Abil- 
ity in Silent Reading illustrates the introduction of supple- 
mentary abilities that tend to destroy the constant func- 
tional relationship between the abilities measured and the 
performance. The pupil is asked to draw pictures. Although 
only very simple drawings are called for, all the pupils do not 
interpret this request in the same way. Some try to pro- 
duce a performance which possesses merit as a drawing. 
Others make only very crude pictures. As a result, a vari- 
able factor is introduced. General intelligence tests which 
require reading introduce a variable ability in the pri- 
mary grades, since some of the pupils have not learned to 
read. / 
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Direct measurement is to be preferred whenever it is feas- 
ible. In this case, the exercises call only for the functioning 
of the abilities to be measured. However, it is seldom possible 
to realize direct measurement. Most educational measure- 
ments are semi-direct. A semi-direct performance may 
bear a constant relation to the ability being measured, but 
such a relation cannot be assumed. Its existence must be 
demonstrated experimentally. In a few tests, the abilities 
which function in the production of the performance do not 
include those which we desire to measure, and the measure- 
ment is called indirect. It is, however, possible that, even 
in the case of such indirect measurement, a constant func- 
tional relation may exist between the performance and the 
abilities which it is desired to measure. [If it exists, it is a 
coincidence and must be demonstrated experimentally. 

Controlling other x-factors. The other z-factors are con- 
trolled by the general structure of the exercises and test, 
and by the directions which are devised for its administra- 
tion. Since tests are designed to be used by different ex- 
aminers at different times, it is, of course, impossible to de- 
vise a measuring instrument which will be absolutely “‘fool 
proof.” Examiners who are not in sympathy with the use 
of educational tests or who are inclined to be careless will, 
as a result, introduce variable factors. But it is possible to 
formulate directions to examiners sufficiently complete and 
sufficiently clear so that examiners who wish to control such 
variable factors as explanations, timing, and manner of pre« 
senting the test, will be successful to a large extent. Con« 
trol of these factors will, in turn, tend to control the effort 
the pupil makes and his emotional status. When pupils 
are not accustomed to educational tests, or the exercises are 
unusual in structure, second-trial scores will be materially 
larger than first-trial scores, especially in the case of rate 
tests. Third-trial scores will be slightly in excess of second- 
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trial scores, but the frequent use of educational tests tends to 
reduce the differences between the scores yielded by succes- 
sive trials. The effect of acquaintance with the type of ex- 
ercises can be partially eliminated by the use of a fore-exer- 
cise, or preliminary test. The use of a printed copy of the 
test for each pupil makes the form of presentation more uni. 
form for all pupils. It is not in the power of the test-maker 
to eliminate deliberate coaching. 


2. Requirement for unrestricted functioning 


The type of ability to be measured. If the performance 
is to be a valid index of the ability of pupils it is obvious that 
each pupil to whom the test is administered must be given 
equal opportunity to give expression to his abilities within 
the field of the test. A test-maker needs to be acquainted 
with the nature of the abilities he is attempting to measure. 
If he has to deal with a general or average ability he should 
proceed differently from what he would do if he were devis- 
ing an instrument to measure separately a number of specific 
abilities. He must, also, be guided by the type of mental 
growth he desires to measure. It is also necessary that the 
pupil be afforded an opportunity to produce a performance 
which will reflect the significant characteristics of the ability 
being measured. For example, if the rate of functioning is 
a significant characteristic of this ability, the pupil must be 
given an opportunity to demonstrate his rate of work. 

In certain types of silent reading the rate of reading is a 
significant characteristic of the pupil’s ability. Therefore, . 
tests designed to measure such types of silent-reading ability 
must provide an opportunity for the pupil to demonstrate 
his rate of reading. For certain abilities the quality or ac- 
curacy of the performance is the significant characteristic. 
In other cases the level of difficulty on which the pupil is just 
barely able to work with a specified quality of performance is 
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a significant characteristic.! Therefore, the type of test to 
be constructed depends upon the nature of the abilities to 
be measured and the purpose for which the test is used. A 
performance must be large enough to represent adequately 
a pupil’s ability. A single exercise is not sufficient. When 
a measure of the ability of individual pupils is desired, each 
pupil must be given the opportunity to do several exercises 
which call for the same ability. Except in extreme cases, 
the longer the performance the more truthfully it will rep- 
resent a pupil’s ability. 

Limitations of functioning due to content. The content of 
the test places certain restrictions on the opportunity which 
the pupil has to demonstrate his abilities. He has no op- 
portunity to demonstrate any except those called for by the 
test. If it is not possible to include exercises which call for 
all of the abilities we desire to measure, the test should be 
representative of these abilities. For example, if the pur- 
pose of a spelling test is to obtain a measure of the pupil’s 
ability to spell the most frequently used words of the Eng- 
lish language, the test words should be representative of the 
most frequently used words of the English language. If the 
function of an arithmetic test is to yield measures of a pupil’s 
ability to do addition examples, the test should consist only 
of addition examples, and these should be representative of 
the various types of addition examples which exist. 

Limitations of functioning due to the structure of a test. 
Certain limitations are imposed in the construction of a test 
by the type of structure of the test itself. These may be 
enumerated under the following headings: 

1. Uniform performances. Most uniform tests consist of 
exercises similar in structure and content. This is the case 


1 Difficulty isa characteristic of the thresholds of ability. It tells just 
how far the pupil is able to go in the doing of more and more difficult tasks 
of a given kind with a specific standard of accuracy. 
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in the Courtis Standard Research Tests, Series B. Each 
test of this series is restricted to a single type of example in 
one of the four fundamental operations. The pupil is given 
no opportunity to demonstrate his ability to do other types 
of examples. He is able to demonstrate his ability to do 
only the one type of example. This constitutes a limita- 
tion of the uniform performance. On the other hand, most 
tests of this type are timed so that a measure of the pupil’s 
rate of work is secured. Thus, the pupil is restricted by 
the scope of the test, but is given an opportunity to demon- 
strate his fluency within a limited field. 

2. Scaled performances. A scaled performance test pre- 
sents a variety of exercises to the pupil. Those on different 
levels of difficulty generally differ in structure and content as 
well as in difficulty. Thus, the pupil has the opportunity to 
demonstrate the range of abilities which he possesses within 
the field of the test. In this respect, a scaled performance 
test is superior to a uniform test. On the other hand, a scaled 
performance test is not usually timed. When there is a time 
limit it is generally set so that practically all pupils have 
ample time to do all of the exercises which they are able to do 
correctly. This is equivalent to no time limit at all. Ina 
few cases the time limit is set so that practically no pupilscan 
complete all of the exercises; but this practice is confined for 
the most part to tests of general intelligence. In Gray’s 
Oral-Reading Test the testing is individual, and the pupil is 
timed on each level of difficulty. However, generally, the 
pupil has no opportunity to demonstrate his rate of work on 
a scaled performance test. This is a serious limitation of 
this type of test, when the rate of work is an important 
characteristic of a pupil’s ability. 

The exercises for a scaled test are selected because they 
are found to possess a certain degree of difficulty, rather than 
because they represent certain content. It may, therefore, 
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happen that certain types of exercises are not found in a 
scaled test because they were not found to possess the exact 
degree of difficulty desired. It appears that this happened 
in the construction of Woody’s Arithmetic Exercises, and 
certain important types of examples are not found in these 
tests. Thus, the pupil is deprived of the opportunity to 
demonstrate his ability to do these examples. 

In anumber of scaled performance tests the pupil is given 
only one exercise on each level of difficulty. One trial does 
not afford the pupil an adequate opportunity to demonstrate 
his ability. He either does the one exercise correctly or fails 
completely.! In such a case no reliable index is obtained 
of the pupil’s ability to do exercises on a given level. It 
may be noted that this restriction does not apply if the 
test is used to secure an average measure of the ability of a 
class or larger group. If each of thirty pupils does an ex- 
ercise we have the same amount of data as we have in the 
measurement of the ability of a single pupil when he does 
thirty exercises. In a few scaled performance tests the pupil 
is given more than one exercise on each level of difficulty. 
For example, the Thorndike Visual Vocabulary Scale pre- 
sents the pupil with ten words on each level of difficulty. In 
other cases, the test-makers have provided for computing a 
pupil’s score from his performance on several levels of diffi- 
culty. If it is assumed that the ability measured is general, 
or that only an average measure is desired, this procedure 
tends to compensate for not giving the pupil an adequate 
opportunity for demonstrating his ability for doing exercises 
on each level of difficulty. 

3. Spiral performances. <A spiral performance possesses 
the virtues of both the uniform and the scaled performance. 
The sub-tests may be, and usually are, timed. Hence, the 


1 A few scaled performance tests provide for partial credit for exercises 
partly correct. 


CONSTRUCTION OF EDUCATIONAL TESTS 78 


pupil is given the opportunity to demonstrate his fluency 
on each level of difficulty, and the sequence of levels provides 
the opportunity for demonstrating the range of his abilities. 
If the levels of difficulty are chosen because they call for 
abilities that are educationally worth while, and not merely 
for statistical reasons, the exercises may be made repre- 
sentative of the field. The number of levels of difficulty 
that are necessary to represent satisfactorily the abilities in a 
field depends upon the number and importance of the spe- 
cific abilities in the field. However, time and expense, and 
the difficulties of describing the performances, limit both the 
number of levels and the number of exercises on each level. 

4. Irregular performances. The irregular performance 
test is used because satisfactory uniform exercises are not 
available, or because it is deemed advisable to have certain 
content represented. This type of test may be constructed 
so that the pupil is given the opportunity to demonstrate his 
ability over a considerable range, but, since we do not have 
any satisfactory plan for describing irregular performances, 
this opportunity is limited in significance. Irregular tests 
may be timed; but, since the exercises are not uniform, and 
generally not confined to one topic, the pupil is handicapped 
in demonstrating his fluency. When the irregularities are 
small, the test may be treated as uniform without intro- 
ducing a serious error. Monroe’s Standardized Silent 
Reading Test, Revised, is an illustration of a slightly irreg- 
ular test which is timed. 

5. Cycle performances. The cycle performance is a device 
to approximate certain of the good features of a uniform 
performance, and at the same time allow for the expression 
of a group of abilities rather than of a single ability. This 
device has its greatest use in dealing with a large group of 
important abilities when it is necessary to economize time. 
Of course, the resulting description will permit only general 
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interpretation. Cycle tests are usually timed. Thus the 
pupil has the opportunity to demonstrate his fluency. 

Exercises must not be ambiguous and must be explicit 
in their specifications. If an exercise is not interpreted alike 
by all pupils they will not have equal opportunities to dem- 
onstrate their ability. If the specifications are indefinite, 
different pupils will attempt to give different kinds of per- 
formances. One limitation of questions in which pupils are 
asked to “discuss” or “compare” is the indefiniteness of the 
demands of the questions. In order to be satisfactory for 
testing purposes an exercise should make clear to the pupil 
exactly what he is to do. 


3. Agreement with educational objectives 


Should measure the effectiveness of the instruction. 
Most achievement tests have for their purpose the measure- 
ment of abilities which have been engendered in the pupils 
by the instruction of the school. This means that the abili- 
ties which such a test measures should be included in the 
objectives of instruction. Otherwise, the measurements 
secured by using it will not be a true index of the effective- 
ness of the instruction. Furthermore, the scores cannot be 
used as a basis for planning remedial instruction. For 
example, if a teacher has organized her instruction in arith- 
metic primarily for the purpose of attaining certain objec- 
tives in the solving of problems, the measures resulting from 
the use of a series of tests upon the operations of arithmetic 
will be of little value as an index of the effectiveness of this 
instruction. 

The use of an achievement test, particularly for the pur- 
pose of diagnosis, suggests to both the teacher and the pupils 
that the norms for the test are worthy educational objec- 
tives. In fact, the use of test norms as educational objectives 
has been generally urged. When the norms are used in this 
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way, it is imperative that the content of the exercises and 
the structure of the test should be in agreement with our edu- 
cational objectives. This statement applies only to achieve- 
ment tests. In the case of instruments for measuring gen- 
eral intelligence, the restriction does not apply, because the 
norms for these tests are not intended to be considered as 
educational objectives. 

Exception in the case of aschool survey. For certain pur- 
poses, such as a school survey, it may be appropriate to use 
tests whose exercises and structure are not entirely consist- 
ent with recognized educational objectives. In fact, the 
position may be taken that the most desirable test in such 
a case would be one which discovered what the pupils had 
learned. For survey purposes, it may be important to know 
whether the pupils have acquired abilities which are not in 
agreement with accepted educational objectives. When 
such a condition is found to exist, it indicates an improper 
distribution of teaching time or emphasis. For example, 
if a group of pupils were found to have been trained in doing 
very difficult and intricate exercises in arithmetic or algebra, 
one would be justified in criticizing the school unless it could 
be demonstrated that the achievements were socially worth 
while. This, however, is a special use of educational tests, 
and for this reason the position is taken that in general the 
content of the exercises and the structure of educational 
tests should be consistent with recognized educational ob- 
jectives. 

Prior determination of objectives. At the present time 
our educational objectives have been adequately determined 
for only a few of our school subjects, such as spelling, hand- 
writing, and arithmetic (in part). In many subjects only 
general statements of our objectives are available. Hence, 
it has frequently been necessary for the maker of a test to 
conduct first a rather elaborate investigation to obtain in- 
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formation with reference to these matters as a preliminary 
step in the construction of his test... This preliminary step 
does not always appear in explicit form in the accounts of 
test construction. This is because it is assumed that the 
abilities which the test is designed to measure are included 
in our objectives. For example, our present practice clearly 
indicates that the abilities required in performing the opera- 
tions in arithmetic with integers are included in our objec- 
tives in arithmetic. No extended investigation is required 
to prove this. Therefore, the test-maker in this field may 
proceed at once to construct tests to measure these abilities.” 

In other fields, where our objectives are not so obvious, it 
has been necessary to determine what they are before a satis- 
factory test can be constructed. For example, in the field of 
spelling, a test which would measure the extent to which 
pupils were attaining our objectives was not possible until 
we had defined our objectives. When our objectives in the 
field of spelling were defined as teaching pupils to spell the 
most commonly used words of the English language, and a 
list of these words had been obtained, as in the Ayres Spell- 
ing Scale, test-makers were then in a position to construct 
spelling tests. In the less formal school subjects, such as 
geography, history, physics, and literature, our objectives 
have not as yet been defined in the way required for test 
construction. Until this is done the activities of test-makers 
in these fields will be decidedly limited. 

Relation of a scaled performance test to educational ob- 
jectives. The idea underlying a scaled performance test is 

l Ayres, L. P. A Measuring Scale of Ability in Spelling. Bulletin of 
Education, 189. Russell Sage Foundation, New York City. 

Monroe, WalterS. Report of the Division of Educational Tests for 1919- 
20. Bulletin No. 5, Bureau of Educational Research, University of Illinois. 
The derivation of Monroe’s Standardized Reasoning Tests is given. 


2 Certain limitations must be observed. One must not assume that ex- 
treme types of examples are included in our objectives. 
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that the pupil will advance along the scale of difficulty until 
he reaches a step which he can just do successfully according 
to the standards of success that have been established for 
the particular scale under consideration. These standards 
differ for different tests. When expressed in terms of a per 
cent of perfect accuracy, 50 per cent, 75 per cent, and 
80 per cent have been used. From a statistical point of 
view 50 per cent is to be preferred. When a pupil’s per- 
formance is described in terms of the highest level of diffi- 
culty reached, his score includes no mention of his perform- 
ance on other levels of difficulty. 

In connection with the use of the norms for scaled per- 
formance tests as educational objectives, two criticisms may 
be made. First, a standard of accuracy which is preferable 
for statistical reasons may not meet social approval. In 
arithmetic or spelling, an objective of less than 100 per cent 
accuracy is open to objection. To hold before the pupil an 
objective which requires only 50 per cent accuracy, or even 
75 per cent accuracy, is contrary to our educational theory 
and practice. Second, the norms of a scaled performance 
test suggest that the teacher’s objective should be to train 
the pupils to do more and more difficult exercises because 
of their degree of difficulty.? 

Such an objective is false for two reasons. First, pupils 
are not to be trained to do things just because they are dif- 
ficult, but because they are educationally worth while. A 
very difficult exercise may have a high educational value, 
but, in many cases, an exercise is difficult just because it 
calls for abilities which are not considered an important edu- 

1 This objection to scaled performance tests could be overcome in part by 
adopting a method of scoring by which the credit given for doing an exercise 
correctly would be proportional to its educational importance. This 
would mean that little credit would be given for doing a very difficult exer- 


cise correctly if it was unimportant educationally. However, this feature 
has not yet been incorporated in any of our scaled performance tests. 
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cational objective, and hence have not received emphasis in 
the school. This would be true of many questions in his- 
tory and geography which call for facts of minor importance, 
and also of complicated exercises in arithmetic and algebra. 
In the second place, such an objective does not include the 
idea of training pupils to do things well and rapidly. Train- 
ing for fluency is not equally important in all fields. For ex- 
ample, it is less important in problem solving in arithmetic 
than in the operations. Even in the case of silent reading, 
we may set up objectives which do not include fluency. 
However, when fluency is included in our objectives, the 
norms for a scaled performance test will be inconsistent 
with them. Thus, we may say that, in general, scaled per- 
formance tests as they are usually scored are inconsistent 
with our educational objectives in many fields. 

Incidentally, it may be noted that this type of test implies 
that ability is general rather than specific. It is assumed 
that the same abilities function in the difficult exercises as 
in the easy ones. 


4. Requirements for the administration of a test, and 
the description of the performance 


Elements in a complete performance. From a practical 
standpoint it is necessary that the test be one which can 
conveniently be given. For anything like general use, this 
means that the test must be such that it can be admin- 
istered to groups of pupils, rather than only to individual 
pupils. The time required should not be excessive, and the 
test itself should be inexpensive. No elaborate accessories 
should be required for its administration. 

The complete description of a performance involves giv- 
ing the magnitude of its three dimensions: 

1. The amount, or the rate at which the ability func- 
tioned in producing it. 
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2. Its quality or accuracy. 

3. The type of exercise in response to which it was given. 

Under certain conditions, this third dimension may be 
given in terms of the level of difficulty reached. The descrip- 
tion of a performance with reference to its amount has no 
meaning unless the test is timed. Hence, if it is desirable to 
have a measure of a pupil’s rate of work it is necessary that 
such a time limit be fixed that no pupil can finish or that the 
time used by each pupil be ascertained. It is highly de- 
sirable that the performance be such that its quality or 
accuracy can be objectively described. When a scorer is 
required to exercise judgment, the measures tend to become 
less accurate. For this reason it is frequently necessary to 
reject exercises which are otherwise satisfactory because 
the resulting performances cannot be objectively described. 
The specific requirements from the standpoint of the de- 
scription of the performance depend upon the kind of infor- 
mation the test is designed to yield. For a rate test there 
are certain requirements. For a power test they are 
different. 

The interpretation and use of the scores will be made eas- 
ier if they are expressed from an absolute zero point and in 
terms of a constant unit. We shall then be able to say that 
a score of 24 means twice the ability indicated by a score of 
12, and that a difference of 4 obtained from 9 to 5 is equal to 
the difference of 4 obtained from 31 to 27. These are not 
necessary requirements, but should be classed as desirable. 


5. Control of testing conditions 


Testing conditions are controlled when they are made the 
same for all pupils, even though tested at different times and 
by different examiners. This has been mentioned briefly in 
connection with the requirement of a constant functional 
relationship between the performance and ability. It will, 
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however, be helpful to emphasize the manner in which this 
control is to be secured. The test-maker cannot completely 
control testing conditions, but much can be accomplished 
by appropriate precautions. Attention is called to the fol- 
lowing items: 

Attitude of the examiner. The examiner should maintain 
the attitude that the purpose of giving the test is to secure 
truthful information concerning the abilities of the pupils. 
The performance desired is one that is representative of their 
abilities. If the pupils are excited they are not likely to give 
a performance that is a truthful indication of their ability. 
If the examiner approaches them with an attitude of indiffer- 
ence the pupils are likely to become indifferent. If the 
pupils have received coaching on the particular exercises, 
their performance will not be a truthful indication of their 
abilities. By an appropriate explanation of the test and by 
suggestions the test-maker can do much to engender a proper 
attitude in the examiner. 

Preparations by pupils. The examiner should be directed 
to have the pupils clear off their desks and provide them- 
selves with well-sharpened pencils, unless they are accus- 
tomed to using penand ink. In the lower grades pencils are 
preferable. In the upper grades the use of pencils is op- 
tional. It is well for each pupil to be supplied with an extra 
pencil, in case he breaks the point of the one in use. The 
pupils should have no material on their desks except that 
which they are to use in the test. If they are to make use of 
any paper, they should be provided with one or two sheets 
of plain paper. 

Distribution of test papers. Such a simple thing as the 
distribution of the test papers should not be left to the judg- 
ment of the examiner. The pupils should be cautioned not 
to examine or open the test folders until directed to do so. 
It is generally necessary to give special emphasis to this di- 
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rection. The examiner should not distribute the test papers 
himself. This should be done by two or three members of 
the class. The distribution will be facilitated if the exam- 
iner counts out the number of test papers for each row. It 
is well to have the tests placed on the desk with the back up. 
This will lessen the possibility of some pupils beginning 
work before the signal to begin is given. 

General data. Blanks calling for the age, name, and grade 
of the pupil, and other general data, should be printed in an 
appropriate place. In case the test is one in which the 
pupil is giving his performance on plain paper, he should 
be asked to record this information. The examiner should be 
cautioned to examine the entries of the pupils, particularly 
in the case of the younger pupils. This may be done after 
the test is completed and corrections may be made from the 
teacher’s records. It is very important that the grade and 
age of the pupil be given correctly. Errors in these items do 
not affect the accuracy of the scores, but they may lead to 
erroneous interpretations of the scores. 

Explanation of tests to pupils. The nature of the exer- 
cises which the pupils are to do must be explained to them. 
Sometimes this is by means of a simple description. Some- 
times it is accomplished by giving them a preliminary exer- 
cise of the same general kind. In any case the purpose is 
to have each pupil understand what is expected of him in 
the test. The explanation of the nature of the test should 
include a statement of how the pupil is to do the exercises. 
For example, in the case of arithmetic, a pupil should be in- 
formed whether he is to check each example before proceed- 
ing to the next, or whether he is to try each example only 
once. It is imperative that the same explanation be given 
by all examiners. Hence it is necessary that the test-maker 
provide the exact explanation which is to be given, and that 
each examiner follow closely these instructions. 
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Timing the performance. In case it is desired to secure a 
measure of the rate at which a pupil works it is necessary to 
have a time limit such that no pupil will complete the test. 
It is also necessary that the time limit specified be rigidly 
observed in giving the test. All pupils must begin at a 
given signal, and stop promptly at another signal. It is 
well to have the pupils assume an attitude of attention, such 
as raising the right hand and looking at the examiner, pre- 
liminary to giving the signal to begin. Requiring that pu- 
pils again assume this attitude of attention when the signal 
to stop work is given will ensure that all pupils stop at the 
same time. 

Illustrative test directions. The following directions for 
Monroe’s Silent-Reading Tests, Revised, are given as typical 
of the directions for many tests: 


GENERAL INSTRUCTIONS 


Tell the pupils not to open the test folder until directed to do so. 
Then have two or three pupils who occupy front seats distribute 
the folders with the first page up, placing a copy on the desk of each 
pupil. 

Be certain that each pupil is supplied with a well-sharpened pen- 
cil. It is better to use pencils than pens, although, if the pupils are 
more accustomed to the pen, it may be used. Have the blanks at 
the top of the first page filled in, assisting pupils if necessary. Be 
certain that none of the items are omitted. Be careful that the 
pupil’s age on his last birthday and the date of his next birthday are 
given correctly. 

It is very important that exactly the time specified for each test 
be allowed for it. An allowance of two minutes means exactly two 
minutes, not two minutes and five seconds or ten seconds. It may 
be contended that, if exactly two minutes are allowed, the pupils 
will not be working two minutes because it will take them a little 
time to get started. This may be true, but the important thing is 
that pupils in different schools shall be given the same time allow- 
ance. ‘The only way in which this can be accomplished is for all 
teachers to follow directions exactly. 
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In order to allow exactly the number of minutes specified for the 
different tests, it will be necessary for the person administering the 
tests to have a watch with a second hand. A stop watch is even 
better, although an ordinary watch that has a second hand will be 
satisfactory if a little care is exercised in using it. The best plan is 
for the teacher to notice the position of the second hand when the 
signal to begin work is given and write down this position. When 
the second hand reaches this point again, a mark should be made. 
Do this for each time the second hand reaches this point. 

The following instructions should be read to the pupils. In 
order that they may be read effectively, the teacher should become 
thoroughly familiar with them before giving the test to the pupils. 
The portions of the following directions enclosed in parentheses and 
printed in italics are not to be read to the pupils. They are “stage 
directions” to the teacher. The directions printed in the test folder 
are reproduced. Hence, it will be unnecessary for the teacher to 
refer to it in reading the directions. 


DIRECTIONS TO BE READ TO THE PUPILS 


Below there are three exercises. Under each exercise there is 
a row of words printed in bold-faced type. Each exercise asks a 
question. You are to read each exercise and then answer the 
question by drawing a line under the right word printed in the 
black type. 

Read the following exercises: 

(a) Iam alittle dark-skinned girl. I wear a slip of brown buck- 

skin and a pair of soft moccasins. I live in a wigwam. What 
kind of a girl do you think I am? 


Chinese French Indian African Eskimo 


> 


The answer to this exercise is ““Indian,”’ so draw a line under 
Indian. (See that the pupils draw a line under the word Indian in 
this exercise.) 


(b) Spring is the time for planting seeds. They grow fastest in 
summer. Autumn is the harvest time. When are seeds put 
into the ground? 

Spring Summer Autumn Winter 
The answer to this exercise is “Spring.” Draw a line under 
Spring. 
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(c) In the sunny land of France there lived a sweet, little maid 
named Piccola. Piccola’s father was dead, and her mother 
was very poor. Draw a line under the word below that tells 
in what country Piccola lived. 


Germany Russia France England 


(The correct answer to this exercise is France. Do not tell the pupils 
what the answer to this exercise is until they have had an opportunity 
to study it. Before going on, ask the pupils if they understand what 
they are to do.) 


On the three following sheets there are a number of exercises like 
these to be read and answered. When the signal is given, turn over 
this page and begin. Work rapidly, but remember that your an- 
swers must be right in order to count. Remember that you are to 
draw a line under only one word in each exercise. Also remember 
that this test is on three pages. When you finish one page turn to 
the next. 

Turn to the next sheet, but do not begin work. Attention! Pen- 
cilsup. (Look at your watch and note the position of the second hand.) 
Ready, Go! (Write down the position of the second hand. Allow 
exactly four minutes.) Stop! Attention! Draw a line through 
the number at the left of the line which you were reading when the 
signal to stop was given. 


6. Description in terms of significant dimensions 


This requirement has been discussed incidentally in con- 
nection with the agreement of the norms with educational 
objectives and the description of the performance. There is 
little to add here. The dimensions that are significant in a 
particular case depend upon the use that is made of the test. 
If the test is to be used to secure a diagnosis of pupils in a 
field where the educational objectives are defined in terms of 
fluency, the significant dimensions are rate and accuracy, or 
some combination of them. If, on the other hand, the edu- 
cational objectives are defined in terms of power or “range 
of acquaintance,” the test should provide for the description 
of the performance in terms of the level of difficulty reached. 
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Thus, the test-maker must first define the function of his pro- 
posed test. Users of tests should exercise care in selecting 
tests that are appropriate for their purposes. 

The law of the single variable. The author! of a recent 
test has formulated a principle with reference to the descrip- 
tion of performances which she has called the law of the 
single variable. Briefly stated, this law means that the 
three variables or dimensions must be recognized in describ- 
ing pupils’ performances. If one of these dimensions is not 
constant for a group of pupils its variation must be recorded 
for each pupil in the process of testing, and the scores must 
be interpreted on the basis of the single variable. For ex- 
ample, if the pupils are to be compared with respect to their 
rate of work it must be shown that both the quality and the 
difficulty of the work done were the same for all pupils. 
“‘All the law of the single variable does not permit is the 
attempt to compare combinations of the three variables in 
unknown or varying amounts.” 

There is some implication in the account of the law of 
the single variable that it means that all pupils should be 
forced to give performances which are constant with respect 
to two of the three variables. It is obvious that this inter- 
pretation of the law cannot be justified. The characteristics 
of the abilities which pupils acquire are not restricted to a 
single variable. When a group of pupils are performing in a 
given subject-matter field in the way which is most natural 
for each, large individual differences are exhibited with re- 
spect to these three variables, particularly with respect to 
rate and quality of work. This is true even in the case of 
pupils who have received the same instruction. Some will 
work with emphasis upon rate; others will emphasize accu- 
racy and work more slowly. If a uniform test is used, the 


1 Burgess, May Ayres. Measurement of Silent-Reading Ability. Division 
of Education, Russell Sage Foundation, 1921. 
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difficulty is constant for all pupils. If a scaled performance 
test is used, different pupils will be working upon different 
levels of difficulty after the first few seconds of the time 
allowed for the test. Since some will advance much farther 
along the scale than others, the average difficulty of the 
exercises done will vary from pupil to pupil. In certain 
subject-matter fields the rate of work is relatively unim- 
portant, and even though all pupils do not work at the same 
rate, variations in the rate of work may be disregarded. For 
example, in spelling the rate of work is not considered im- 
portant. The rate also appears to be unimportant in the 
case of painting or drawing when the products are real works 
of art. 

If a variable is controlled by an arbitrary procedure 
which produces unnatural conditions, it appears likely that 
the ability which functions is modified, or factors are intro- 
duced into the testing procedure which produce the same 
result upon the performance. For example, if in the case of 
handwriting all the pupils of the group were forced to write 
at a fixed rate, those who were accustomed to write more 
rapidly or more slowly than this rate would exhibit abilities 
different from those which normally function in their hand- 
writing. Therefore, any attempt to control by arbitrary 
conditions one or more of the dimensions of a pupil’s per- 
formance is likely to introduce serious errors in the resulting 
measures. 

The real point of this law, as indicated above, is that 
measuring instruments must explicitly recognize the possible 
existence of the three variables or dimensions of a pupil’s 
performance. These must be described separately if accu- 
rate interpretation of the scores is to be possible. If a varia- 
ble is omitted in a pupil’s score, it must be shown to have 
been constant in its effect upon the performances of the 
different pupils or to be socially unimportant. When two 
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or more of the variables are combined in a single score, this 
combination is likely to consist of “unknown and varying 
amounts” of the different variables. For example, the 
number of exercises done correctly is frequently used as a 
pupil’s score. This is a combination of the rate and the 
quality of the performance. The combination is in the form 
of a product, but in no case does the score show the relative 
magnitude of the two factors. Hence, in interpreting scores 
which are the number of exercises done correctly, it is neces- 
sary to recognize the limitations of an unanalyzed total. It 
is, however, possible that the practical convenience of hav- 
ing the performance described in terms of a single score 
largely compensates for the loss in precision of the interpre- 
tation. 


7. Equal spacing of exercises on the scale of difficulty 


If the exercises of a scaled performance test are equally 
spaced on the scale of difficulty, certain simplified methods 
of describing the performance may be used. (See page 123.) 
The equal spacing also gives a more random selection of 
exercises on the basis of difficulty. 

Selection of exercises for a test. The requirements dis- 
cussed in the preceding pages should guide one in deciding © 
upon the type of test to be constructed, and indirectly in the 
construction of the exercises. It is now necessary to con- 
sider the procedures to be followed in the selection of the 
exercises required for a particular type of test. This will be 
discussed under three heads. 

1. Content. If a general test is to be constructed, each ex- 
ercise should call for all of the abilities within this field, or 
the group of exercises chosen should call for a random sam- 
pling of them. For example, a spelling test which is designed 
to yield a general or average measure of the pupil’s spelling 
ability cannot include all words which the pupil should be 
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able to spell. Therefore, it should consist of words which have 
been selected so that they constitute a random sample of the 
words included in our objective in spellmg. ‘The same situa- 
tion should exist with reference to general tests in other 
fields, such as geography, history, silent reading, and so 
forth. One procedure that has been followed in order to se- 
cure exercises that constitute a random sampling is to take 
the subject-matter as presented in a textbook and select 
items appearing at regular intervals. For example, in the 
case of spelling, we have had spelling tests constructed by 
taking the first defined word appearing on the even-num- 
bered pages of a dictionary. The same method has been 
used for selecting the words for a vocabulary test. Another 
method which has been followed has been to compare a 
number of textbooks and to use the items which were com- 
mon to the several texts. Sometimes this procedure has 
been supplemented by securing the opinions of competent 
judges with reference to the importance of the exercises. 

One author has selected exercises for a language test on 
the basis of pupil needs.! Performances of pupils given 
under normal conditions were examined to ascertain the 
grammatical errors which they made, and exercises were 
then constructed which called for the correction of the errors 
occurring most frequently. A somewhat similar method was 
followed by Ayres in constructing his spelling scale. He 
selected words occurring most frequently in the correspond- 
ence and other writings of adults. They are the words of 
the English language most frequently used in writing. 

2. Suitableness for testing purposes. It frequently happens 
that an exercise will prove on trial to be unsuitable for test- 
ing purposes. Except in the construction of a scaled test, 
very easy exercises and very difficult exercises are to be 


1 Charters, W. W. “The Construction of a Language and Grammar 
Scale’; in Journal of Educational Research, vol. 1, p. 249 (April, 1920). 
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avoided. ll exercises which are confusing or ambiguous 
should likewise be avoided. Occasionally exercises must be 
rejected because the performances secured by them cannot 
be conveniently described. An experienced test-maker 
will be able to avoid including a large number of unsatisfac- 
tory exercises in the preliminary list; but the only certain 
procedure is to have the exercises given to several hundred 
pupils and to analyze the responses that are obtained. 

3. Difficulty. We are concerned with the difficulty of ex- 
ercises in both uniform and scaled tests. In the former, if 
the exercises have not been constructed so that equal diffi- 
culty may be assumed, it is necessary to have the exercises 
given to a large number of pupils so that those which are 
done correctly by the same per cent of the pupils may be se- 
lected. In the construction of a scaled performance test it 
is necessary to select from a preliminary list those exercises 
which will form a difficulty scale. This requires that the 
difficulty of each exercise in the preliminary test be deter- 
mined and expressed in terms of a convenient unit. In do- 
ing this, the exercises are given to a large number of pupils, 
and the per cent of correct responses for each exercise is cal- 
culated and translated into the corresponding difficulty 
value. 

A number of test-makers have followed a very elaborate 
procedure in constructing such a scale, but some recent 
studies of the scoring of scaled performance tests indicate 
that serious errors would not be introduced if the procedure 
were greatly simplified. For example, the difficulty of each 
exercise might be computed from all grades combined in- 
stead of for each grade separately. Another possible pro 
cedure is to base the difficulty values upon the results secured 
from only one grade. Either method will eliminate the calcu- 
lation of the inter-grade interval, which will greatly shorten 
the process for tests designed to be used in a sequence of 


92 THEORY OF EDUCATIONAL MEASUREMENTS 


several grades. We shall describe the complete process, but 
any one contemplating the construction of a scaled perform- 
ance test may well consider the advisability of abbreviating 
the procedure. 

Assumptions underlying the construction of scaled tests. 
The makers of scaled tests have made two assumptions. 
First, it is assumed that when an unselected group of pupils, 
such as those belonging to a given school grade, is distrib- 
uted according to a given ability, a normal distribution is 
secured. (See Fig. 3.) Supplementary to this assumption 
it is implied by the procedure of this group of test-makers 
that this assumption holds for the particular group of pupils 
to whom the exercises were given in the course of the con- 
struction of the test. The second assumption is that the 
variability of this normal distribution is the same for succes- 
sive school grades. Both of these assumptions appear to be 
approximately in agreement with available data. Further- 
more, they do not appear to be inconsistent with a priori 
deductions. It is true that some have contended that pupils 
found in the upper grades and in the high school have been 
selected from a total population in such a way that a skewed 
distribution would be obtained. This contention must be 
supported by additional proof before it can be accepted. In 
any case, it does not appear that much would be gained by as- 
suming a skewed distribution instead of anormal distribution. 

An additional assumption appears to be implied by this 
method of test construction. It appears to be assumed that 
the giving of the exercises in the preliminary list without 
any time limit provides adequate opportunity for pupils to 
demonstrate their ability.! This means that in judging 


1 Occasionally instead of allowing unlimited time for the entire list of 
exercises submitted to pupils, they are given a definite time allowance for 
each exercise. This time allowance is intended to be generous and to give 
& pupil the opportunity to do the exercise in case he has the ability. 
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the abilities of pupils it is sufficient to recognize only the 
quality of their work, and that their rate of work is not sig- 
nificant. As we have pointed out in other connections, the 
rate of work is significant in the case of some abilities. In 
the case of others it is relatively unimportant. This assump- 
tion is not violated in such cases. 

Selection of exercises for a scaled test. In the construc- 
tion of a scaled test there are three rather distinct steps. 
First, it is necessary to compile a preliminary list of exer- 
cises which will include a wide range of difficulty, and which 
will be representative of the field of the proposed test. An 
effort should be made to have all levels of difficulty repre- 
sented, particularly the levels at the extremes of the scale. 
Next, this preliminary list is given to a large number of pu- 
pils in the grades for which the final test is intended. On 
the basis of results secured a tentative list of exercises for 
the final test is selected. 

In making this selection, two criteria are observed. First, 
only those exercises are chosen which were done correctly by 
a gradually increasing per cent of the pupils as one proceeds 
from the lower to the higher grades. If an exercise is done 
correctly by a higher per cent of the pupils in the lower 
grades than in the higher grades, it is rejected as being un- 
suitable for testing purposes. The other criterion requires 
that the exercises selected be equally spaced on the scale of 
difficulty. It frequently happens that gaps are found so 
that the complete realization of this requirement is not pos- 
sible. In such cases it is necessary to interpose exercises in 
the gaps found to exist in the preliminary list. These exer- 
cises, of course, must be given to the same group of pupils or 
to a similar group in order to ascertain the per cent of correct 
responses that may be expected. In extreme cases it will be 
necessary to revise the preliminary list and give the entire 
set of exercises to a new group of pupils. 
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The third step involves the evaluation of the exercises of 
this tentative list in terms of their difficulty, and the arrange- 
ment of them in the form of a scale. The evaluation of the 
exercises involves the translation of the per cents of pupils 
doing them correctly into difficulty values, the determination 
of the inter-grade interval, and the location of the zero point. 

Relation between per cent of correct responses and diffi- 
culty. Ability and difficulty are correlative. The doing of 
an exercise having great difficulty demands the possession 
of a high degree of ability. The pupil who is just able to do 


NUMBER 
OF PUPILS 


fe) ABILITY OF PUPILS 
DIFFICULTY OF EXERCISES 


Fig. 3. A GrapaicaL REPRESENTATION OF A NoRMAL DISTRIBUTION 
SHowInG RELATION BETWEEN NuMBER oF PuPpiLs AND ABILITY 


an easy exercise gives evidence of possessing only a small 
degree of ability. In representing the relation between per 
cent of correct responses and difficulty we usually repre- 
sent the degrees of ability on a horizontal line. Until a 
more suitable zero point is established, it is customary to 
choose the average as zero. A point to the right of this 
zero represents a degree of ability greater than the average 
of the group of pupils being considered. A point to the. 
left represents a less degree of ability. Vertical distances 
represent the number of pupils who possess exactly the de- 
grees of ability represented on the base line. Our assump- 
tion, concerning the distribution of ability in an unselected 
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group gives a symmetrical curve which is called the normal 
curve. (See Fig.3.) 

In describing a position on the base line of this normal 
curve a measure of the variability or spread of the curve is 
used. Both the median deviation (P.E.) and the standard 
deviation (o7) have been used for this purpose. If we take 
the degree of ability represented by +1.00 P.E.— i.e., the 
degree of ability represented by a position 1.00 P.E. to the 
right of our assumed zero — we find that it is possessed or 
exceeded by 25 per cent of the pupils. Similarly, if we 
take the degree of ability represented by —1.00 P.E. it is 
possessed or exceeded by 75 per cent of the pupils. 

The normal curve defines a fixed relation between the two 
variables, ability and number of pupils. Hence, it is possi- 
dle to calculate the per cent of pupils who possess or exceed 
any given degree of ability. The per cent of pupils who do 
an exercise correctly is simply the per cent of pupils who 
possess or exceed the degree of ability required to do that 
exercise. Since ability and difficulty are correlative terms, 
we can calculate the degree of difficulty of an exercise from 
the per cent of correct responses secured for it. It is only 
necessary to determine the per cent of pupils to the right of 
various points on the base line of the normal distribution. 
Tables have been prepared which give the per cent of pupils 
to the right of various points along the base line of the nor- 
mal curve. In Table I we give the distances from the mid- 
point of the normal distribution corresponding to various 
per cents of correct responses. For example, there are 10.56 
per cent of all pupils to the right of +1.250. The per cent 
to the right of — 4.00 P.E. is 99.65. Also, by means of this 
table, the difficulty of any exercise can be determined with ref- 
erence to the midpoint of the normal distribution when we 
know the per cent of correct responses received for it. It is 
only necessary to find, in the column of “per cent correct,” 
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TABLE I 


RELATION BETWEEN PER CENT OF CORRECT RESPONSES 
AND DIFFICULTY OF EXERCISES 


Difficulty Difficulty 
Per cent Per cent 
eorrect S.D. P.E. comnece S.D. P.E. 
(c) (c) 

99.999971 —5.00 —7.41 99.29 —2.45 —3.63 
99 .999963 —4.95 —7.34 99.18 —2.40 —8.56 
99 .999952 —4.90 —7.26 99.06 —2.35 —3.48 
99 .999938 —4.85 -—7.19 98.93 —2,30 —3.41 
99.99992 —4.80 —7.12 98.78 —2.26 —3.34 
99.99990 —4.75 —7.04 98.61 —2.20 —3.26 
99.99987 —4.70 —6.97 98.42 —2.15 —3.19 
99.99983 —4.65 —6.89 98.21 —2.10 —-3.11 
99.99979 —4.60 —6.82 97.98 - 2.05 —3.04 
99.99973 —4,55 —6.75 97.72 —2.00 —2.97 
99 .99966 —4.50 —6.67 97.44 —1.95 —2-89 
99 .99957 —4.45 —6.60 97.18 —1.90 —2.81 
99.99946 —4.40 —6.52 96.78 —1.85 —2.74 
99 .99932 —4.35 —6.45 96.41 —1.80 —2.66 
99.99915 —4.30 —6.37 | 95.99 —1.75 —2.59 
99.9989 —4.95 —6.30 | 95.54 —1.70 —2.52 
99.9987 —4.20 —6.23 95.05 —1.65 —2.45 
99.9983 —4.15 —6.15 94.52 —1.60 —2.37 
99.9979 —4.10 —6.08 93.94 —1.55 —2.30 
99.9974 —4.05 —6.00 93.382 —1.50 —2.22 
99.9968 —4.00 —5.93 92.65 —1.45 —2.15 
99.9961 —3.95 —5.86 91.92 —1.40 —2.08 
99.9952 —3.90 —5.78 91.15 —1.85 - 2.00 
99.9941 —3.85 —5.71 90.382 —1.380 —1.93 
99.9928 —3.80 —5.63 89.44 —1.25 —1.85 
99.9912 —3.75 5.56 88.49 —1.20 —1.78 
99.989 —3.70 —5.49 87.49 —-1.15 —1.70 
99.987 —3.65 —5.41 86.43 —1.10 —1.63 
99.984 —3.60 —5.34 85.31 1.05 —1.56 
99.981 —3.55 —5.26 84.18 —1.00 —1.48 
99.977 —3.50 —5.19 82.89 — .95 —1.41 
99.972 —3.45 —5.11 81.59 — .90 —1.83 
99.966 —3.40 —5.04 80 .23 — .85 —1.26 
99.960 —3.35 —4.97 78,81 — .80 —1.19 
99.952 —8.30 —4,.89 77.34 — .75 -1.11 
99.942 —3.25 —4.82 75.80 — .70 —1.04 
99.9381 —38.20 —4.74 74.22 — .65 — .96 
99,918 —3.15 —4.67 72.57 — .60 — .89 
99,903 —3.10 —4.60 70.88 — .55 — .82@ 
99,886 —3.05 —4.52 69.15 -— .50 — .74 
99.865 —3.00 —4.45 67.36 — 45 — .67 
99.84 —2.95 —4.37 65.54 — .40 — .50 
99.81 —2.90 —4.30 63.68 — .85 — .52 
99.78 —2.85 —4.3 61.79 — .80 — .44 
99.74 —2.80 —4.15 59.87 — .25 — .87 
99.70 —2.75 —4.08 57.93 — .20 — .80 
99.65 —2.70 —4.00 55.96 — .15 — .22 
99.60 —2.65 —3.938 53.98 — .10 — .15 
99.53 —2.60 —3.85 51.99 — .05 — .07 
99.46 —2.55 —3.78 50.00 00 + .00 

+ .07 
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TABLE I (continued) 
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Difficulty 

DS. P.E. 

(c) 

+2.60 +3.85 
+2.65 +3.93 
+2.70 +4.00 
+2.75 +4.08 
+2.80 +4.15 
+2.85 +4. 23 
+2.90 +4.30 
+2.95 +4.37 
+3.00 +4.45 
+8.05 +4.52 
+38.10 +4.60 
+3.15 +4.67 
+3.20 +4.74 
+3.,25 +4.82 
+3.30 +4,89 
+3.35 +4,97 
+3.40 +5.04 
+38.45 +5.11 
+3.50 +5.19 
+3.55 +5.26 
+3.60 45.34 
+3.65 +5.41 
+3.70 +5.49 
+8.75 +5.56 
+38 .80 +5.63 
+3.85 +65.71 
+3.90 +5.78 
+3.95 +5.86 
+4.00 +5.93 
+4.05 +6.00 
+4.10 +6.08 
+4.15 +6.15 
+4.20 +6.23 
+4,95 +6.30 
+4.30 +6.37 
+4.35 +6.45 
+4.40 +6.52 
+4.45 +6.60 
+4.50 +6.67 
+4.55 +6.75 
+4.60 +6.82 
+4.65 +6.89 
+4.70 +6.97 
+4.75 +7.04 
+4.80 +7.12 
+4.85 +7.19 
+4,90 +7.26 
+4.95 +7.34 
+5.00 +7.41 
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the given per cent of correct responses and read the corre- 
sponding o or P.E. value.1_ Most test-workers have used 
P.E. as the unit, although there is no reason why it should 
be preferred. The two units sustain a constant relation to 
each other: 1.00 P.E. equals .67450. 

Finding the inter-grade interval. The preliminary list 
of exercises should be given to groups of pupils in all grades 
in which the final test is to be used. Makers of this type 
of measuring instrument have generally calculated the 
difficulty of the exercises separately for each grade, and 
then combined the different results secured. The average 
ability of the different grade groups is not the same. Gen- 
erally, there is a gradual increase in the average ability 
from a lower to a higher grade. It has been assumed that 
the variabilities of the distributions of the different grades 
were equal, which means that the difficulty values deter- 
mined from different grade groups are expressed in terms 
of acommon unit. However, the difficulty value of an ex- 
ercise in the fifth grade cannot be combined with the diff- 
culty value of an exercise in the seventh grade until these 
values are expressed with reference to the same zero point. 
To do this requires the determination of the inter-grade in- 
terval, or the distance between the assumed zero points of 
the distributions of ability in the successive school grades. 

Three methods have been proposed for making this deter- 
mination: 

1. Exercise method. 
2. Quartile method. 
8. Distribution method. 


1 Several different forms of tables have been used in expressing the rela- 
tion between the per cent of correct responses and the difficulty of exercises. 
In order to economize space, most tables have not been extended beyond 50 
per cent; but the normal distribution which we get in such a table can easily 
be adapted so as to yield determinations for per cents above 50. Some 
tables have been expressed in terms of per cents of incorrect responses 
rather than in terms of correct responses. 
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1. Exercise method. Since each exercise is given to pupils 
in different school grades, the difference in its value for two 
successive grades, when both values are expressed as a dis- 
tance from the midpoint of the distributions of the respec- 
tive grades, is easily determined. If we assume that these 
values expressed with reference to the absolute zero point 
would be identical, the difference between them is the differ- 
ence between the assumed zero points, or the midpoints of 
the distributions of these two grades. The method may be 
illustrated by example 10 of Woody’s Addition Scale. It 
was found to be 1.629 P.E. below the midpoint of the third- 
grade distribution, and 2.397 P.E. below the midpoint of the 
fourth-grade distribution. The difference (inter-grade in- 
terval) is .768 P.E. In this way the inter-grade interval 
between two successive grades may be found for each exer- 
cise that was attempted by pupils in both grades. 

The determinations of the inter-grade interval secured in 
this way will not be equal. In general, the more difficult the 
exercise is for the pupils of the grades concerned the greater 
the inter-grade interval will be. For this reason, it has been 
proposed that, instead of taking the average of all the deter- 
minations of the inter-grade interval, the determinations 
should be weighted so as to give the greater weight to those 
determinations secured from moderately easy or moderately 
difficult exercises. 

2. Quartile method. By this method the performance of 
each pupil is described in terms of the number of exercises 
done correctly. The scores thus obtained tend to form a 
normal distribution for each grade. The median and the 
quartile deviations are calculated for each distribution. 
The difference between the medians of two successive grades 
is divided by the average of the corresponding quartile de- 
viations. For example, if the median for the second grade 
is 6.8, and the one for the third grade is 14.5, the difference is 
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7.7. If the corresponding quartile deviations are 2.7 and 
3.0, the average is 2.85. Using this as a divisor into 7.7, the 
inter-grade interval is found to be 2.7 quartiles or P.E. 

3. Distribution method. The calculation of the inter- 
grade interval by this method is based upon the fact that 
the distribution for one grade overlaps those of the adjoin- 
ing grades. If the number of examples done correctly is 
used as the pupil’s score, there will be some pupils in the third 
grade who have higher scores than the median score of the 
fourth grade, and others who are below the median score of 
the second grade. The per cent of pupils who have scores 
between the median for their grade and the median for an- 
other grade can be used as the basis for calculating the inter- 
grade interval. Such a per cent is merely the per cent of 
pupils between the midpoint of a normal distribution and a 
point to the right or left whose distance is to be determined. 
_ Jf the point is to the left, i.e., if it is the inter-grade interval 
between a lower grade, 50 should be added to the per cent of 
pupils and then the inter-grade interval may be read directly 
from Table I. If the distance is to the right of the zero 
point, i.e., if it is the inter-grade interval between a higher 
grade, the per cent of pupils must be subtracted from 50 
before Table I can be used. 

In applying the distribution method it is not necessary 
that adjacent grades be used. Any two grades between 
which there is overlapping may be used. The interval ob- 
tained when the two grades are not adjacent is not the inter- 
grade interval, but rather the sum of the inter-grade inter- 
vals between the two grades used. For example, if the third 
and fifth grades have been used, the interval will be from the 
median of the third grade to the median of the fifth grade. 
Knowing this distance, and having calculated the inter- 
grade interval from the third to the fourth grade, we can 
obtain a determination of the inter-grade interval from the 


CONSTRUCTION OF EDUCATIONAL TESTS 101 


fourth to the fifth grade. Such secondary determinations 
are not as significant as the primary determinations and 
should be given less weight in calculating the average inter- 
grade interval. 

Combining the results of the three methods. Some test- 
makers have used all three methods and have calculated a 
weighted average of the three determinations. Others have 
used only one method. It appears that the exercise method 
is considered to yield the most significant results and, hence, 
the determination by this method has been given more 
weight than the other determinations in calculating the 
average. 

Locating the zero point. In all of the preceding work the 
difficulty values of the exercises have been expressed as dis- 
tances from the midpoint of the grade distributions. With 
the inter-grade intervals known, it is possible to express all 
distances from a common zero point, such as the median of 
the third-grade distribution.‘ If this were done some dis- 
tances would still be negative, and such a scale would not be 
as convenient to use as one which involved only positive 
quantities. Obviously, negative quantities can be elimi- 
nated by choosing the zero point sufficiently far to the left and 
expressing all values with reference to it. There is, however, 
one disadvantage in locating the zero of a scale in this way. 
We are unable to say that an example that has a scale value 
of 5 is just half as difficult as one that has a scale value of 10, 
because, if we had located the arbitrary zero 4 units farther 
to the left, the scale values of these two exercises would have 
been 9 and 14, respectively. Had the zero point been lo- 
cated 4 units to the right, their scale values would have been 
1 and 6, respectively. Before a scale value of 10 can be in- 
terpreted to mean twice the difficulty of a scale value of 5, 
both must be expressed with reference to an absolute zero 


1 This assumes that the exercises were given to pupils of the third grade. , 
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point. Sucha point would represent not any of the ability 
in question. 

The methods used in locating the zero point are more or 
less arbitrary, and due caution must be exercised in inter- 
preting the zero points chosen as meaning not any of the 
ability in question. A common method is to note the per 
cent of pupils failing to do a single exercise correctly in the 
lowest grade in which the exercise has been given. These 
pupils are considered not to possess any of the ability being 
measured. The remaining per cent of the pupils (100 minus 
the per cent doing no exercise correctly) possess some ability; 
i.e., they are above the absolute zero point. By referring to 
Table I the distance of this zero point from the midpoint 
of the distribution may be ascertained. The same method 
may be applied to the distribution for any grade which in- 
cludes zero scores. The zero point determined in this way 
approaches a true absolute zero point only when exercises 
calling for the least definable amount of ability are included 
in the list submitted to the pupils. 

Woody has also calculated the zero point by dividing the 
median number of examples done correctly by the quartile 
deviation of the distribution. This method, however, ap- 
pears to be less satisfactory than the one described above. 

Determining the final scale values of the exercises. 
Knowing the inter-grade interval and the location of the 
zero point, it is now possible to calculate the final scale value 
for each exercise. This value will denote its distance from 
the zero point chosen. It is calculated from the determina- 
tions in the several grades. These, when corrected for the 
inter-grade intervals, will differ rather widely in some cases. 
This is most likely to happen when the exercise is either 
very easy or very difficult for the pupils of a given grade. 
Such determinations should be given less weight than 
those secured when the exercise is done correctly by from 25 
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to 75 per cent of the pupils. Woody gives a value double 
weight if the exercise is less than 1 P.E. distant from the me- 
dian achievement of a grade distribution, single weight if it 
is more than 1 P.E., but less than 3 P.E. distant from the 
median achievement, and he does not consider it at all if it is 
more than 3 P.E. distant from the median. The weighted 
average which he obtained from this method is taken as the 
general or final value of the exercise. 

After the values of all of the exercises have been deter- 
mined, a group of exercises for the final scale should be se- 
lected. If possible those exercises should be chosen which 
are equally spaced on the scale of difficulty. Unless a large 
number of exercises have been evaluated, it may happen 
that no such group can be selected. If it is deemed impera- 
tive to have a group of exercises that are equally spaced on 
the scale of difficulty, it will be necessary to prepare another 
list of exercises and repeat the procedure just described. 


QUESTIONS AND TOPICS FOR DISCUSSION 


1. What requirements should govern the construction of educational 
tests? 

2. What requirements of test construction appear to have been recog- 
nized by the makers of the following tests: 

a. Woody’s Arithmetic Exercises? 

b. Buckingham’s Spelling Scale? 

c. Ayres’s Scale for Measurement of Ability in Spelling? 

d. Burgess’s Picture Supplement Scale for Measuring Ability in 
Silent Reading. 

e. Courtis’s Standard Research Tests in Arithmetic, Series B? 

3. Did the authors of the above tests overlook any essential require- 
ments? If so what ones? 

4. One of the requirements to be observed in test construction is that 
all pupils be given an opportunity to demonstrate their abilities in 
the field of the test. Is this requirement adequately provided for in 
the following tests: 

a. Courtis’s Standard Research Tests in Arithmetic, Series B? 
b. Woody’s Arithmetic Exercises? 

c. Cleveland Survey Tests in Arithmetic? 

d. Monroe’s Standardized Silent-Reading Test? 
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e. Burgess’s Picture Supplement Scale for Measuring Ability in 
Silent Reading? 
f. Charters’s Diagnostic Tests in Language and Grammar? 

5. Distinguish between a ‘‘power test” and a “‘speed test.”” (See McCall, 
W.A., Correlation of Some Psychological and Educational Measurements. 
Teachers College Contributions to Education, No. 79, pp. 44-53.) 

6. What types of tests may we recognize from the standpoint of structure? 

7. Does the available evidence justify us in assuming that indirect meas- 
urement is possible? (Buckingham, B. R. ‘‘Proposed Index to Effi- 
ciency in Teaching U.S. History”; in Journal of Educational Research, 
vol. 1, p. 161 (March, 1920). a 
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CHAPTER V 
DESCRIPTION OF THE PERFORMANCES OF PUPILS 


A quantitative scale is required for description. A scale is 
necessary for quantitative description. It is a part of every 
measuring instrument. In some it appears in very obvious 
form. In others it is implied in the structure of the test, the 
rules for scoring, and the method of computing scores. The 
complete description of a pupil’s performance requires a 
quantitative statement of its quality and of the rate at 
which it was produced, together with the specifications of 
the exercises in response to which the performance was 
given. These specifications are expressed in various forms. 
For the Courtis Standard Research Tests in Arithmetic, 
Series B, a statement of the structure of the exercises is suf- 
ficient. For example, in the case of the addition test it is 
sufficient to say that each exercise consists of three columns 
of figures, with nine figures to the column. In other fields 
the specifications cannot be given in such a simple form. 
One of the forms in which the specifications are sometimes 
given is in terms of the difficulty of the exercises of the test. 

The method of describing a performance varies for each of 
the three characteristics, and also for different types of per- 
formances. Some tests provide a separate description for 
each of the three characteristics. Other measuring instru- 
ments combine two and, in a few cases, all three in a single 
score. It will, therefore, be necessary to consider separately 
the description of each of the three characteristics with ref- 
erence to the different types of performance. After this we 
shall note some of the methods of description which combine 
two or more of the characteristics in a single score. 
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Rate of uniform performances. In order that a measure 
of the rate at which a performance has been produced may 
be secured, two requirements are necessary. First, the per- 
formance, or some representative portion of it, must be 
timed; and, second, the timed portion of the performance 
must be of such a nature that it can be divided into equal 
work units,! or into exercises consisting of known multiples 
of a convenient work unit. For group testing the plan gen- 
erally used, and the most convenient one, is to obtain only 
the performance that pupils are able to produce within a 
suitable time interval. Another plan which is sometimes 
used is to have each pupil indicate the portion of the per- 
formance which he completes within a given time interval. 
This second plan is not very satisfactory. In individual 
testing it is possible to time each pupil on either the entire 
performance or a portion of it, as may be desired. The 
number of unit performances given within the time interval 
constitutes a quantitative statement of the pupil’s rate of 
work. For convenience the rate is frequently expressed in 
terms of the number of units of work done within a unit of 
time, such as a minute or second. 

The structure of the test usually determines what the 
work unit shall be. In some subject-matter fields, such as 
the operations of arithmetic, it is possible to construct the 
exercises so that they form approximately equal units of 
work. The structure of a number of tests is such that it is 
not convenient to divide the pupil’s performance into work 
units which are approximately equal. For example, in the 
case of silent reading, the most convenient work unit to use, 
and the one which is generally used, is the word. It is ob- 
vious that one word is not necessarily equal to another word. 
However, the variations of particular words from an “aver- 


1 “Equal work units” may be defined as divisions of a test which in gen- 
eral are done in equal time intervals and with a constant quality. 
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age word” are chance variations, and when the performance 
is sufficiently long the word does constitute a satisfactory 
unit for measuring the rate of work. A similar situation ex- 
ists in the case of handwriting. The letter is usually taken 
for the unit of work, but one letter is not equal to another 
letter. 

Rate of non-uniform performances. In irregular tests the 
exercises do not represent equal units of work. However, 
when the irregularities are not large and are distributed ina 
random manner, and the number of units is sufficiently large, 
the error introduced by considering the exercises to represent 
equal work is slight. Thus the rate may be described in 
terms of the number of exercises attempted, as in the case of 
uniform performances. 

In the case of scaled performances, or performances with 
marked irregularities, the rate is probably not constant 
throughout the performance. One would expect it to be 
more rapid on the easier exercises, and less rapid on the more 
difficult ones. Thus the exercises do not represent equal 
work units, and it is difficult, if not impossible, to divide the 
performance into equal work units. For this reason the use 
of scaled performance tests or of tests with marked irregu- 
larities should be avoided, if possible, when the rate of work 
isimportant. Several plans for weighting the exercises have 
been proposed, but they are unsatisfactory for securing a 
true measure of the rate of the performance. Certain sys- 
tems of weighting which have been used will be described in 
the dicussion of the description of the quality of non-uni- 
form performances. 

Two measures of quality. In some subject-matter fields, 
such as arithmetic or spelling, we are accustomed to judge 
the quality of a performance as either right or wrong. Im- 
perfections are definite errors. In such subject-matter fields 
quality becomes accuracy. In other subject-matter fields — 
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imperfections are not errors in the sense that eleven would be 
an erroneous sum for 7+5. In English composition, for 
example, there are some imperfections which are errors, but 
a particular composition may in many ways fall short of per- 
fection, not because of errors, but because certain qualities 
are not present in the fullest possible degree. Frequently 
these qualities are subtle. What is true of English compo- 
sition and other similar performances is true to some extent 
of silent reading, oral reading, and the answers to certain 
kinds of questions, but in these subjects exercises have been 
constructed and rules for scoring devised so that the de- 
scription of performances is essentially in terms of accuracy. 

When a pupil’s performance on an exercise cannot be 
classified as right or wrong, or at least as right, partially 
right, or wrong, quality becomes a matter of merit rather 
than of accuracy. It will therefore be necessary to make 
certain differentiations in our discussion of the description 
of the quality of performances. We shall consider first 
quality when it is defined as accuracy, and second when it 
is defined as merit. The description of the latter will be 
discussed in the next chapter. 

Accuracy of uniform performances. The usual form of 
describing the accuracy of a uniform performance requires 
that it be divided into equal units of work. The accuracy of 
the performance is then expressed as the per cent of these 
units in which a certain standard of accuracy has been at- 
tained. The expression of the accuracy as a per cent is not 
essential except for purposes of comparison when all pupils 
do not attempt the same number of exercises. If sufficient 
time is allowed, so that all pupils have had the opportunity 
to attempt all of the exercises which they are able to do cor- 
rectly, we may describe the quality of their performances 
in terms of the number of units done correctly. However, 
the measure is probably more easily understood when ex- 
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pressed as a per cent. It should be noted that when ex- 
pressed as a per cent the accuracy is an index of the average 
quality of: the performance. It must not be interpreted to 
mean that throughout the entire performance the pupil was 
accurate only to the extent indicated by the accuracy score. 
As a matter of fact his performance was perfect for some 
units and for other units was incorrect. 

The scale of accuracy. The scale of accuracy when ex- 
pressed as a per cent is from 0 to 100. This, however, does 
not mean that a pupil’s per cent of accuracy may always be 
any number between 0 and 100. If the performance consists 
of 10 units and partial credits are not given, the possible 
measures of accuracy are 0, 10, 20, 30, ete. If the perform- 
ance consists of 20 units the possible measures of accuracy 
are 0, 5, 10, 15, 20, 25, ete. 

Standards of accuracy. An accuracy score of zero per 
cent does not necessarily mean that the pupil did absolutely 
nothing correctly. For example, in the addition test of the 
Courtis Standard Research Tests, Series B, an answer is 
counted as entirely wrong if one figure is incorrect. If a 
pupil has no answer entirely correct, his accuracy score is 
zero, even though all figures except one may be correct in 
every example. For the same reason, an accuracy score 
of 40 per cent does not mean that in the remaining 60 per 
cent of the units of work the pupil did nothing correctly. It 
simply means that in 60 per cent of the units of work the 
pupil failed to attain the required standard of accuracy. 
Thus the meaning of an accuracy score depends upon the 
standard of accuracy which has been set for judging the 
units of the performance. 

The units of most performances may be considered to con- 
sist of elements or of opportunities to make errors. For ex- 
ample, an exercise in the addition of integers consisting of 
three columns of figures of nine figures each involves the 
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addition of twenty-seven numbers, plus two numbers to be 
carried. Each figure must be read and the answer must be 
written. Such a unit may be analyzed in several ways, but 
it is obvious that' any form of analysis will give a large 
number of opportunities for making an error. The stand- 
ard of accuracy depends upon the length of the unit exercise, 
i.e., upon the number of opportunities for error, and upon 
the penalty to be charged against the pupil for making an 
error. 

For purposes of illustration, take two exercises in the addi- 
tion of integers: 

(A)6 (B)45724635 

3 2S IT S 2-42 


Both of these examples consist of addition combinations 
whose sums are lessthanten. Hence no carrying is involved 
in either example. Assuming that for the same pupil the 
chance of an occurrence of error is the same for all of these 
combinations — i.e., that he knows them equally well —the 
opportunities for error are eight times greater in example 
B than in example A. Consequently, according to the usual 
rule for scoring such examples, success on example A does 
not mean the same as on example B. Success on example B 
requires a higher quality of work. By increasing the num- 
ber of opportunities for error it would be possible to set an 
example containing a sufficient number of elements so that 
it would be done correctly by practically no pupil of the 
elementary school. This would probably be true if the num- 
ber of elements were increased to 1000. Although the pupil 
might “know” all of these addition combinations, which 
are the simpler addition combinations, human fallibility 
would eventually enter in to produce an error. 

The standard of accuracy also depends upon the penalty 
which is levied against a performance for an error. Con- 
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sider two tests upon the multiplication combinations. Each 
test consists of 100 combinations. In the first test each 
combination is considered a unit exercise. A pupil who 
makes twelve errors will be penalized twelve units or com- 
binations, and the accuracy of his performance will be 88 
per cent of perfection, defining perfection to mean the writ- 
ing of 100 combinations without error. The second test is 
divided into unit exercises of five combinations each. The 
test will consist of twenty of these units. Suppose a pupil 
in doing these twenty exercises (100 combinations) writes 
twelve combinations incorrectly. These errors, if we as- 
sume a random distribution of the combinations within the 
test, are probably distributed rather uniformly over these 
100 combinations. The maximum number of wrong exer- 
cises would be twelve and the minimum three. The true 
number is probably between these two extremes, although 
nearer twelve than three. 

The usual plan of scoring is to penalize a pupil for an entire 
unit if that unit contains one error. The same penalty 
is levied against his performance if the unit contains more 
than one error. According to this plan, if all of the errors 
were concentrated in three units, the per cent of accuracy 
would be 85. If the twelve errors were distributed so that 
there was only one in a unit exercise, the per cent of accu- 
racy would be 40. These are the extreme possibilities. If 
the errors were uniformly distributed over the perform- 
ance, the per cent of accuracy would fall between these 
extremes, but in no case would it be equal to that on the 
other test. A pupil’s accuracy score on this second test 
would differ from his accuracy score on the first test, not by 
reason of the actual number of errors made, but by reason 
of the penalty which was levied for an error. 

Arbitrary standards of accuracy. In arithmetic or spell- 
ing the standard of perfect accuracy is usually adopted. It 
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has the approval of social usage. But other standards could 
be set up. For example, in arithmetic an answer could be 
counted correct if only one figure was wrong, or the spelling 
of a word could be considered correct if the sound of the 
letters used agreed with the pronunciation of the word. It 
would be absurd to set up such standards in the operations 
of arithmetic or in spelling except for a special investigation, 
but for tests in other fields similar standards are frequently 
established. In oral reading Gray has set a standard which 
involves giving full credit when only one error is made. In 
scoring answers to questions that require pupils to formulate 
answers in their own words, considerable variation is gen- 
erally allowed in the answers that are accepted as correct. 
Thorndike does this in his Scale Alpha 2 for the Under- 
standing of Sentences. Partial credit is sometimes given 
for answers which are judged to be partially right. 

Two types of uniform performances. In such subject- 
matter fields as the operations of arithmetic, where the ex- 
ercises of a test may be constructed so that they contain the 
same number of elements, a uniform performance is gener- 
ally one in which the exercises consist of the same number 
of elements and consequently imply the same standards of 
accuracy. In other types of subject-matter fields differ- 
ent exercises may consist of different numbers of elements, 
although they are equally difficult. Asaresult the stand- 
ards of accuracy are likely to be different. The per cent of 
correct responses obtained for an exercise depends upon two 
things: the number of opportunities for error, and the chance 
of occurrence of error in anelement. Fora given per cent 
of correct responses these two factors vary inversely. The 
greater the number of elements the less the chance of the 
occurrence of error in an element, and conversely. We may, 
therefore, have a uniform performance in which the units , 
contain the same number of opportunities and the chance of 
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the occurrence of error is the same for all units. We may, 
however, have a uniform performance in which the number 
of elements per unit is different, and the chance of occurrence 
of error in an element likewise different. For practical pur- 
poses we do not need to distinguish between these two types 
of uniformity. 

Equivalent standards of accuracy necessary for compara- 
ble measures. In our discussion of standards of accuracy 
we showed that for the same performance different descrip- 
tions would be obtained, depending upon the standard of 
accuracy used. It therefore follows that measures of accu- 
racy are not comparable unless the standards of accuracy on 
which they are based are equivalent. The standards of 
accuracy will be equivalent when the unit exercises upon 
which both are based contain the same number of elements, 
and the penalty levied against the performance for the oc- 
currence of error is the same. It is only when these condi- 
tions are fulfilled that comparisons of accuracy scores have 
significance. For example, on the basis of the four tests of 
the Courtis Standard Research Tests, Series B, as they are 
usually scored, it is impossible to make any inferences con- 
cerning the relative accuracy of pupils in the different tests. 
It has been inferred from the scores resulting from the use 
of this series of tests that addition was a more difficult pro- 
cess than subtraction, because the average per cent of accu- 
racy is greater for subtraction than for addition. Even a 
casual examination of these two tests will reveal that the 
standards of accuracy by which the performances of pupils 
are judged on these two tests are not equivalent. The pen- 
alty for error is much more severe in the case of addition. 

Another condition must be fulfilled if small differences 
between measures of accuracy are to be significant. If a 
performance consists of three units, the per cent of accu- 
racy is either 0, 383, 663, or 100. If a performance consists 
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of 10 units the possible per cents of accuracy are 0, 10, 20, ete. 
If a performance consists of 50 units the possible per cents 
of accuracy are 0, 2, 4, 6, 8, etc. In comparing the per cent 
of accuracy on a performance of 50 units with the per cent of 
accuracy of a performance consisting of 3 units, it is clear 
that no significance can be attached to small differences be- 
tween the two measures. A similar statement may be made 
with reference to the comparison of individual scores with 
norms. 

In order that small differences between the measures of ac- 
curacy may be significant it is necessary that both of the per- 
formances consist of a large number of units. This require- 
ment is generally violated for some pupils by allowing the 
same time interval for all pupils when a timed performance 
is secured. Since some pupils work much more rapidly than 
others, the performances of the slower pupils will contain 
only afew units. In order to secure performances which are 
approximately equal in length, it will be necessary to modify 
the timed performance tests which are now commonly used. 
One plan that has been used by certain test-makers is to 
have the pupil mark the place reached at specified intervals. 
Courtis has done this in his Silent Reading Test No. 2. 
This plan is only partially satisfactory. 

Accuracy of non-uniform performances. In describing 
the accuracy of a non-uniform performance, it appears rea- 
sonable that more credit should be given for doing a very 
difficult exercise correctly than for giving the correct re- 
sponse to a very easy one. Several attempts have been 
made to devise a method of weighting the exercises which 
would compensate for the differences in their size and diffi- 
culty. In most cases exercises of different levels of diffi- 
culty imply different standards of accuracy. When the 
answer to a long and intricate example in arithmetic is called 
wrong because a single figure is incorrect, the standard of 
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accuracy is distinctly different from that which is used in 
judging the correctness of a simple example. This makes it 
difficult to devise a procedure which will yield a valid de- 
scription of the average accuracy of the performance and 
which may be easily interpreted. Several of the proposed 
plans .are described below. After weights have been as- 
signed to the exercises according to a given plan, the sum 
of the weights, or values, of the exercises done correctly is 
usually taken as the pupil’s score. In case the test has been 
timed so that different pupils have attempted different 
numbers of exercises, such a score is a combination of rate 
and quality. 

Plans for weighting exercises of non-uniform perform- 
ances. 1. Fordyce, in constructing his silent-reading test, 
assigned values to the exercises that are proportional to the 
square root of the per cent of incorrect responses. He states 
that this is in accord with the law of least squares. This is 
an effort to give more credit for doing a difficult exercise 
than for doing an easy one. 

2. One writer has proposed that we assign values propor- 
tional to the number of incorrect responses. ! 

8. Weights have been assigned to exercises proportional 
to the degree of difficulty expressed in terms of the mean 
deviation (P.E.), or standard deviation with the zero point 
fixed arbitrarily.? 

4. In the derivation of the Kansas Silent Reading Tests, 
F. J. Kelly obtained for each exercise the average number of 
seconds per correct answers. Some of the exercises were 
longer than others and hence required more time. Some 
were short but difficult in the sense that the per cent of cor- 


1 Brooks, 8. S. ‘Getting Teachers to Feel the Need for Standardized 
Tests”; in Journal of Educational Research, vol. 11, pp. 431 (June, 1920). 

? Monroe, Walter S. Report of Division of Educational Tests for 1919-20. 
University of Illinois Bulletin, vol. xvi, No. 31 (January 24, 1921). - 
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rect responses was small. The length of time required to do 
the exercise and the degree of difficulty were combined by 
this method.! 

5. It has been pointed out that the allowance of more 
credit for difficult exercises than for easy ones does not in- 
flict upon a pupil the proper penalty for failing on a simple 
exercise. It has, therefore, been suggested that the best 
description of a pupil’s performance will be expressed in 
terms of a system of credits for correct responses, and pen- 
alties for incorrect responses. ? 

6. The regression equation is probably the most scientific 
method for weighting exercises. This method requires that 
we have an independent criterion of the ability being meas- 
ured. This is difficult to secure. In addition, the use of 
the regression equation involves an intricate statistical pro- 
cedure. 

7. The above plans of weighting, except the last, are ex- 
plicitly for the purpose of adjusting the credits given to the 
difficulty of the various exercises. These proposals imply 
that the significance of a pupil’s performance on an exercise 
is a function of the difficulty of the exercise. In defining our 
educational objectives we generally assume that the im- 
portance of topics is determined by their usefulness. Many 
very difficult exercises are unimportant because they are not 
useful. For this reason it has been proposed to weight exer- 
cises on the basis of their social importance. According to 
this proposal the doing of a very difficult exercise might add 


1 Kelly, F.J. ‘Kansas Silent Reading Tests”’; in Journal of Educational 
Psychology, vol. vu, pp. 62-80 (February, 1916). 

Monroe, Walter S. ‘‘Monroe’s Silent Reading Tests”; in Journal of 
Educational Psychology, vol. 1x, pp. 303-12 (June, 1918). 

2 Minnick, J. H. An Investigation of Certain Abilities Fundamental to 
the Study of Geometry. Philadelphia: University of Pennsylvania, 1918. 

3 Kelley, T. L. Vocational Guidance, p.95. Teachers College Contribu- 
tions to Education, No. 71. 
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little to a pupil’s score. This method demands a criterion 
of social usefulness. Because this is difficult to secure, prac- 
tically no use has been made of the method. 

Scores based upon unweighted exercises. Any plan of 
weighting adds considerably to the task of constructing the 
test and to the scoring of the test papers. For this reason it 
has been proposed that the unequal difficulties of the exer- 
cises be disregarded, and that scores be calculated just as 
though the exercises were equal in difficulty. Although 
this procedure appears to be illogical, it has been found that 
the correlation between weighted and unweighted scores is 
extremely high. Charters dropped the weights from the 
exercises in his language and grammar tests because he felt 
that the differences in the scores obtained by the two meth- 
ods were not sufficient to justify the added labor involved 
in computing the weighted scores. 

Description of a scaled performance. Since a scaled per- 
formance is not usually timed, the consideration of the de- 
scription of the rate of such a performance may be omitted. 
The description of quality is implied in the statement of the 
highest level of difficulty reached with the fixed standard of 
accuracy. Hence the description of a scaled performance is 
confined to a single score which is intended to define the step 
of the scale which the pupil was barely able to reach with a 
fixed standard of accuracy. 

The theory of this type of measure is that the pupil is pre- 
sented with a series of exercises or groups of exercises which 
gradually increase in difficulty. The situation has been 
likened to such athletic events as the pole vault, high jump, 
and so forth. As the pupil advances along the scale he will 
finally reach the level where he does not have the ability 
(power) to do the exercises with the established standard of 
accuracy. When this ideal condition exists a pupil’s ability 
(power) is described in terms of difficulty of the highest step 
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of the scale which he does satisfactorily. However, the typ- 
ical pupil will not do all exercises or groups of exercises up 
to a certain point with a degree of accuracy up to or above 
the standard set, and fail to attain this standard of accuracy 
on all succeeding exercises. He will fail to attain the stand- 
ard accuracy for certain exercises, and then attain or ex- 
ceed it for more difficult ones. For a considerable range 
of the scale he will alternate between success and failure. 
These fluctuations complicate the problem of describing a 
pupil’s performance on tests of this type. It is necessary to 
secure a performance on a series of levels of difficulty in the 
neighborhood of the theoretical level, on which a pupil just 
barely attains the standard of accuracy. Of course, in 
group testing, where a single test is applied to all pupils 
under the same conditions, it is necessary to have a consid- 
erable range of difficulty in order that each one may give a 
performance on an appropriate series of levels of difficulty. 

Relation of accuracy to difficulty. The study of the per- 
formances of pupils on successive levels of difficulty has 
indicated that the relation between the quality or accuracy 
of the performance and the levels of difficulty is represented 
by the ogive curve. The equation of this curve is: 


pues 


This equation will be recognized as an integral of the equa- 
tion of the normal distribution: 


dy sae 

pe e 2a2 

dz 
In the above equation y stands for the quality of the pupil’s 
performance; i.e., the per cent of exercises done correctly on 
a given level of difficulty. The other variable, z, stands for 
the difficulty of the given group of exercises. In Table I 
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(see page 96) we have given the values of y which correspond 
to values of zx between —5o and +50. When these values 
are represented graphically an ogive curve is obtained. In 
Fig. 4 the zero point has been chosen arbitrarily at —8. 
With this modification it is a representation of Table I. 
Standards of accuracy with respect to which level of 
difficulty is expressed. This relationship between quality 


ACCURACY 


8 4 5 6 7 8 9 10 MN tens 
DIFFICULTY 


Fie. 4. Tar Octve Curve, SHowine THE RELATION BETWEEN Ac- 
CURACY AND Dirricutty. Tuis 1s A GRAPHICAL REPRESENTATION OF 
Taser I wits THE Zero Pornt at — 8. 


of performance and level of difficulty appears to hold for 
both individual pupils and for groups of pupils. In Fig. 4 
and in Table I it is clear that near 100 per cent accuracy, or 
near zero accuracy, the change in accuracy corresponding to 
a unit change in difficulty is very small. For example, in 


DESCRIPTION OF PUPIL PERFORMANCE 121 


Table I the change in accuracy between a level of difficulty 
of —5o and a level of difficulty of — 40 is .003171 per cent; 
between — 40 and — 3a it is .1318 per cent. On the other 
hand, change from — 10 to 0a is 34.13 percent. A similar con- 
dition exists on the higher levels of difficulty. If the fixed 
standard of accuracy is chosen near 100 per cent, it is ob- 
viously necessary to have a very large number of exercises on 
each level of difficulty, or to have some means for ascertain- 
ing very precise measures of the accuracy of a pupil’s work. 
In view of the variable character of pupil performances a high 
degree of precision in expressing the degree of accuracy 
cannot have much significance. For these reasons, it is 
necessary to choose a standard of accuracy for which the 
ratio of the rate of change of accuracy to the rate of change 
of difficulty is near a maximum. This ratio is at a maxi- 
mum for 50 per cent accuracy. It recedes from this maxi- 
mum rather slowly for small distances on either side of a 
standard of 50 per cent. Pedagogical reasons may be given 
for using astandard of accuracy higher than 50 per cent. 
Thus some makers of tests have chosen a standard of ac- 
curacy of 80 per cent. In one case a standard of accuracy 
of 75 per cent has been used. From the standpoint of meas- 
urement a standard of 50 per cent is to be preferred. 
Estimating the level of difficulty corresponding to a given 
standard of accuracy. Because of the irregularities in the 
accuracy of pupil performances, and the imperfections of 
difficulty scales, it is generally necessary to estimate the 
level of difficulty on which the established standard of accu- 
racy would be attained. This estimate is based upon the 
quality of the pupil performance on the levels of difficulty 
near this point on the scale. If we assume that for a given 
pupil or a given group of pupils the relation between the 
accuracy of their performances and the levels of difficulty 
on which these performances are given is that represented 
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by the ogive curve, it is possible to estimate, from a pupil’s 
accuracy on known levels of difficulty, the theoretical level 
of difficulty on which the established standard of accuracy 
would be obtained. 

Suppose that on the level of difficulty 7.10 a pupil has 
rendered a performance which is 80 per cent accurate. The 
ogive curve which corresponds to the degree of ability pos- 
sessed by this pupil must pass through the point whose co- 
ordinates are 7.10 and 80. In Fig. 4 this is curve A. We 
wish to know the level of difficulty on which he would be most 
likely to render a performance which is 50 per cent accurate. 
If a horizontal line is drawn to represent 50 per cent of accu- 
racy it will cut this ogive curve at 0. A perpendicular from 
this point will cut the horizontal axis, the point which rep- 
resents the level of difficulty on which a performance 50 per 
cent accurate may be expected. This point is 8.0. This 
method is entirely general. It is only necessary to deter- 
mine the position of the particular ogive curve correspond- 
ing to the ability of a particular pupil. This is done by lo- 
cating the point whose codrdinates are the per cent of accu- 
racy and the difficulty on a level of the test. 

In calculating a pupil’s score on a scaled performance 
test, the level of difficulty on which he will meet the stand- 
ard of accuracy is estimated from his performances on sev- 
eral levels of difficulty. The usual procedure is to use all 
performances on all levels except those for which the per 
cent of accuracy is approximately either zero or 100. The 
pupil’s score is the average of these several estimates. This 
graphical procedure is obviously too tedious for practical 
use. Thorndike has constructed tables for estimating the 
level of difficulty on which his standard of accuracy (80 per 
cent) is attained on his Visual Vocabulary Scales, and also 
on his Scale Alpha 2 for the understanding of sentences. 
His tables are, however, not satisfactory for computing in- 
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dividual scores. T. L. Kelley has prepared a more elaborate 
table to be used in computing such scores on Alpha 2.1 
Kelley’s shorter method of estimating scores. T. L. 
Kelley ? has proposed a “simplified method” for computing 
the scores of individual pupils. His procedure assumes 
that the differences between successive levels of difficulty 
are equal. If this assumption is not made the procedure is 
greatly complicated. According to his method, the pupil’s 
performance upon each level of difficulty is described with 
reference to accuracy. If the performances on each level 
consist of the same number of units, this description may 
be given in terms of the number of units done correctly. 
In case a performance which ranges in quality from perfect 
accuracy to zero accuracy has not been obtained, it is neces- 
sary to estimate extensions of the pupil’s performance to 
include these limits. These estimates are based upon the 
relation which is represented by the ogive curve. The differ- 
ences between the accuracy on successive levels of difficulty 
are then calculated. Because pupils tend to be erratic in 
their performancessome of these differences may be negative. 
These differences are next added algebraically, beginning 
with the ones for the lower levels of difficulty, until the sum 
is half of the magnitude of the scale of accuracy. For exam- 
ple, if the accuracy of the performance has been described 
in terms of per cent, half of the magnitude of the scale of 
accuracy would be 50. If each level of difficulty consists of 
10 units, and the accuracy has been described in terms of the 
number of units done correctly, half of the scale would be 5. 
The level of difficulty reached when the partial sum of the 
differences is equal to the half sum is the level of difficulty 


1 Kelley, T. L. ‘“Thorndike’s Reading Scale, Alpha 2, Adapted to In- 
dividual Testing”; in Teachers College Record, vol. xvm, pp. 253-60 (May, | 


1917). 
* Kelley, T. L. “A Simplified Method of Using Sealed Data for Pur+ 


poses of Testing”; in School and Society, vol. tv, pp. 34-37. 
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on which the pupil may be expected to work with an accu- 
racy of 50 per cent. This procedure will then be recognized 
as simply calculating the median of the distribution of the 


TABLE II. KELLEY’S SHORTER METHOD OF CALCULATING 
DIFFICULTY SCORES APPLIED TO TRABUE’S LANGUAGE™- 
COMPLETION TEST 
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differences. If some standard of accuracy other than 50 
has been chosen, the procedure must be changed accordingly. 
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The above method has been modified by Kelley by using 
the average instead of the median of the differences. This 
has been done in Kelley’s arrangement of Trabue’s Comple- 
tion-Test Exercises for individual testing. His procedure is 
illustrated in Table II. In the first column the levels of 
difficulty of the scale are given. In the second column the 
record of a pupil is given in terms of the number of exercises 
done correctly on each level. Beginning with level 12 the 
scores are estimates made in accordance with the relation 
expressed by the ogive curve. In column three the decrease 
in number right is given for successive levels of difficulty. 
In column four the midpoints of the successive levels of diffi- 
culty are given. The last column contains the products of 
columns three and four. To obtain the pupil’s score, the al- 
gebraic sum of these products is divided by 10, the number 
of units on each level of difficulty. The quotient is 12.8, 
which defines the level of difficulty on which this pupil would 
be most likely to give a performance which was exactly 50 
per cent accurate. It should be noted that the decrease in 
the accuracy of the pupil’s performance from level 10 to 
level 11 is —1. This decrease is simply considered as a 
negative quantity and does not in any way complicate the 
procedure. 

Van Wagenen’s shorter method of estimating scores. 
Van Wagenen! has further simplified the procedure for cal- 
culating a pupil’s score on a scaled performance test by pro- 
viding tables from which the estimates may be read directly. 
His tables have been constructed for 10 exercises or tasks on 
each level of difficulty, and for levels representing integral 
difficulty values. Similar tables could be constructed for 
other scales. When a pupil’s performance includes a level 


1 Van Wagenen, M.J. “Table for Computing Mean Individual Scores 
in Educational Scales”; in Teachers College Record, vol. xxt, pp. 441-51 
(November, 1920). See also his reading scales in history and general science. 
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of difficulty on which he has made no errors, the procedure 
for calculating his score is simple. Take the lowest level 
of difficulty for which the number of errors is 10. From Van 
Wagenen’s table one can obtain the estimated level of diffi- 
culty on which the number of errors would be 5 or 50 per 
cent of the number of possible errors. If there is no level 
for which the number of errors is 10 — i.e., if the pupil has 
done some of the exercises correctly on the highest level of 
the scale — the table provides for making a similar estimate 
from his record on this level. This estimate is not the pu- 
pil’s score. It must be corrected for all errors made on 
lower levels of the scale. The method of doing this is to find 
the sum of the errors made on these levels. Divide this 
sum by 10, and subtract the quotient from the estimate.! 
The remainder is the pupil’s score; i.e., the theoretical level 
of difficulty on which he would just be able to do 50 per 
cent of the exercises correctly. If the pupil fails to do cor- 
rectly all of the exercises on the lowest level of the test, it is 
necessary to estimate the number of additional errors he 
would have made if the test had been extended downward 
to such a level. These estimates are given in a table and 
must be subtracted from the corrected estimate obtained 
above. 

Description of scaled performances in terms of the num- 
ber of exercises done correctly. The plan of describing 
a pupil’s total performance in terms of the level of difficulty 
on which he has just barely attained a given standard of 
accuracy requires that the performance on each level of 
difficulty be described with reference to accuracy.2. To do 
this with even moderate precision requires that a pupil do 


1 This division by 10 may be avoided by multiplying the scale values by 
10, and thus making the unit .1 P.Z. This Van Wagenen has done in his 
reading scales for history and general science. 

» When there are the same number of exercises on each level this descrip- 
tion may be given in terms of the number of correct answers. 
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several exercises on each level of difficulty or that a system 
of partial credits be devised. The inclusion of a large num- 
ber of exercises in a test will obviously require a large ex- 
penditure of time in testing. In order to reduce the test- 
ing time test-makers in certain fields have included only 
one exercise on each level of difficulty. This has been done 
by Woody, Starch, Hotz, and others. Even when the pupil 
is asked to do several exercises on each level of difficulty, 
as in the Thorndike Scale, Alpha 2, some users of such tests 
have described the pupil’s performance in terms of the 
number of exercises which he does correctly. 

It may be contended that this procedure amounts to 
giving equal credit for exercises which explicitly vary 
widely in difficulty. However, if the exercises are equally 
spaced along the scale of difficulty, and the easiest one has 
a scale value of 1, the number of exercises done correctly 
will give an approximate description of the level of difficulty 
reached. Even when these two conditions are not met, 
the description may be sufficiently accurate for practical 
purposes. 

Relation between number of exercises done correctly and 
score in terms of level of difficulty. Wan Wagenen’s General 
History Scale A and General History Scale B were given to 
a group of pupils. The test papers were scored both accord- 
ing to the directions given by Van Wagenen and also in terms 
of the number of exercises answered correctly.! Scale A 
was given to 36 pupils and Scale B to 34. The coefficient of 
correlation between the regular scores and the number 
of exercises done correctly was found to be .87 + .028 for 


1 In this test the pupil is asked to check a list of statements giving those 
that are in agreement with a paragraph that he has read. A pupil may make 
errors in two ways. He may check a statement that is not true, and he may 
fail to check a statement that is true. He may also do an exercise correctly 
in two ways, either by checking true statements or by not checking state- 
ments which are untrue. 
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Scale A, and .91 + .021 for Scale B.! For Scale A the prob- 
able error of estimate in terms of the regular score is 2.9.? 
In terms of the number of exercises done correctly it is 2.1. 
For Scale B the probable errors of estimate are 2.6 and 2.1, 
respectively. Thirty-one of these pupils took both Scale 
A and Scale B. The coefficient of correlation between the 
two sets of the Van Wagenen scores is .78 + .04. The prob- 
able variable error of measurement? is 1.2. Although the 
probable error of estimate is approximately twice as large as 
the probable error of measurement, it is not likely that the 
description of a pupil’s performance on this test, in terms of 
the number of exercises done correctly, introduces serious 
errors. Considering the magnitude of the probable variable 
error of measurement, it would appear that one would be 
entirely justified in using the simpler method of describing 
the pupil’s performance. 

The sum of difficulty values used as scores. It has been 
proposed that the sum of the difficulty values of the exer- 
cises done correctly should be taken as a pupil’s score on a 
scaled performance test. This procedure results in a pupil 
receiving more credit for doing a difficult exercise than 
for doing an easy one. In the Virginia Survey a spelling 
test was constructed by taking one word from each of the 
twenty consecutive columns of the Ayres Spelling Scale. 
Thus words which gradually increased in difficulty were se- 
cured. These words were weighted by assigning a value of 
one to the first, of two to the second, and so on, with the 
twentieth word having a value of twenty. The pupil’s score 
was taken as the sum of the weights of the words spelled 

1 For an explanation of the statistical terms used here and in the follow- 
ing pages, see Chapters IX, XII, and XIII. 

2 The formula for the probable error of estimate is 

P.E. Bete = 6745 6g \/1 — Pig 
* This is computed by the formula, P.E. y= .6745 6/1 —ryo 
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correctly. Under the writer’s direction, this spelling test 
was given to 306 fifth-grade pupils in one school system. 
The performances of the pupils were described according to 
the rules used in the Virginia Survey. They were also de- 
scribed by giving one credit for each word spelled correctly. 
This assumes that the same credit should be given for the 
different words. The coefficient of correlation between 
weighted scores and unweighted scores for this group was 
.96 + .0038. In the same school system the test was given to 
184 seventh-grade pupils, and the coefficient of correlation 
between the weighted and unweighted scores for this group 
was found to be .97 = .0050. 

The coefficient of correlation expresses only a general 
relationship. These coefficients approach perfect correla- 
tion closely, and indicate that there is close agreement be- 
tween the two sets of scores. We may also express the close- 
ness of this relation in terms of the probable error of estimate. 
For the fifth-grade group it is 10.4 in terms of the weighted 
scores and .57 for the unweighted scores. The standard 
error of estimate for the unweighted scores means that, if 
the weighted scores were reduced to the same scale as that of 
the unweighted scores, so that equivalent scores would be 
identical in magnitude, fifty per cent of the pupils would 
receive scores which differ from each other by no more than 
.57 of a word. In the seventh grade the standard errors 
of estimate are 10.2 and .50. Thus, it is seen that in this 
case it makes very little difference which of these two 
methods of scoring is used. The error introduced by giving 
the same credit for all words is slight. Incidentally, it may 
be noted that the reliability coefficient of this test is prob- 
ably no greater than .75 for these groups of pupils. For 
this degree of reliability, which is high for a twenty-word 
test, the probable error of measurement is approximately 
10.0 for the weighted scores, and 0.5 for the unweighted. 
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Thus, the difference between the two methods of scoring is 
no greater than the probable error of measurement. 

Combined scores. For certain purposes it is desirable to 
have the pupil performance described in terms of a single 
number. This necessitates that we select only one dimen- 
sion of his performance, or that we combine the descriptions 
of the dimensions which are recognized. The number of 
exercises done correctly has been used as the pupil’s score on 
rate tests. On such tests this score is a combination of rate 
of work and accuracy of performance. For example, a pupil 
may make a score of ten exercises right by doing ten exercises 
with 100 per cent accuracy. He may also make a score 
of ten exercises right by doing twenty exercises with 50 
per cent accuracy. The number of exercises done cor- 
rectly is essentially the product of the rate of work and 
of the accuracy when the latter is expressed as a per cent. 
When the number of exercises done correctly is used to de- 
scribe a pupil’s performance on a scaled test, his score is a 
combination of quality and difficulty. If the scaled test is 
timed, the rate of work will be included in the combination. 
In his Oral Reading Test, W. S. Gray uses a plan for securing 
the combination of three dimensions. A pupil is given a 
single score which is a function of his rate of reading, the 
quality of his reading, and the difficulty of the exercises 
which he has read. 

A combined score is useful when a summary or general 
description of the pupil’s performance is desired. However, 
the number of exercises right is an unanalyzed score and for 
this reason its interpretation is limited. On a rate test we 
cannot know whether the pupil’s deficiency is in rate or in 
quality. When difficulty is introduced the situation is even 
more complex.! 

Accuracy when number of answers is limited. When the 


1 See page 87 for the “‘law of the single variable.” 
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number of possible answers is limited there is a chance that 
a pupil will give the correct answer by chance. This is espe- 
cially true in the case of exercises which permit of only two 
answers. In general a pupil who knows nothing about the 
exercises will be able to answer half of them correctly by 
guessing. It, therefore, becomes necessary to make allow- 
ance for this condition in describing the quality of a pupil’s 
performance on such a test. When there are as many as 
four or more possible answers the effect of this condition 
is slight and is usually neglected in computing the pupil’s 
score. The following formula gives a method for making 
allowance for a limited number of possible answers. The 
number of possible answers is represented by n, the number 
of answers correct by c, and the number of answers wrong 
by w. A pupil’s score equals 


CcC—W 


n—1 


When there are only two possible answers, as in a “true- 
false” test, this formula becomes “the number right minus 
the number wrong.” 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1. What information must be given in order to describe a pupil perform- 
ance completely? 

2. What are the requirements for describing the rate of a pupil’s per- 
formance? ‘ 

3. What is the usual scale of accuracy? 

4. How is accuracy defined? What are the requirements for describing a 
performance with reference to accuracy? 

5. What determines the standards of accuracy? 

6. In a scaled performance a typical pupil does not do all exercises cor- 
rectly up to a certain point and then fail to do any exercises beyond that 
point. A typical performance is one in which the pupil fails upon some 
exercises and then does correctly other exercises which are more diffi- 
cult. Explain why this happens. 

7. Upon what basis can the use of the number of exercises done correctly 
as a pupil’s score on a difficulty scale be justified? 
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8. What is the application of the “‘law of the single variable” with refer- 
ence to the description of a pupil performance? 
9. In terms of what dimensions is the pupil’s performance described in 
the following tests: 
1. Gray’s Oral-Reading Test? 
2. Woody’s Arithmetic Exercises? 
3. The Cleveland Survey Tests in Arithmetic? 
4, The Courtis Standard Research Tests in Arithmetic, Series B? 
5. The Van Wagenen Reading Scales in History and General 
Science? 
. Courtis’s Silent-Reading Test, No. 2? 
7. Minnick’s Geometry Test? 


a 
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CHAPTER VI 


DESCRIPTION OF PERFORMANCES OF PUPILS: 
QUALITY SCALES 


Description of a performance when quality does not mean 
accuracy. When a pupil’s performance or some division of 
it cannot be described as right or wrong, or at least as par- 
tially right, it is necessary to construct a scale which can be 
used in describing its quality. This condition exists in hand- 
writing, English composition, hand-sewing, and drawing. 
A quality scale is simply a collection of sample perform- 
ances of the type to be described, which exhibit varying 
degrees of quality from the lowest quality with which we 
have to deal up to the highest such quality. The samples 
are arranged in order of increasing quality, and usually the 
quality of each is described numerically. It is desirable to 
have the samples selected so that the differences in quality 
between successive samples are approximately equal. It is 
also desirable to have the numerical designations of the de- 
grees of quality express the amount of quality measured 
from a zero which means no quality at all. 

A quality scale of this type is used by matching the per- 
formance to be described with the sample of the scale which 
most nearly resembles it in quality. The decision which 
the scorer reaches is only an opinion, and we find that differ- 
ent persons match the same performance with different scale 
samples. Hence this matching is a subjective process; i.e., 
it depends upon the person who does it. Appropriate train- 
ing will materially reduce the subjectivity of the process.! 


1 Thorndike, E. L. “‘ Teachers’ Estimates of Specimens of Handwrit- 
ing”; in Teachers College Record (November, 1914). 

Gray, C. T. “The Training of Judgment in the Use of the Ayres 
Scale of Handwriting,” in Journal of Educational Psychology, vol. v1, 
pp. 85-97 (February, 1915). 
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Construction of a quality scale. The essential steps in the 
construction of a quality scale are three: 

1. Collecting sample performances which represent as 
wide a range of the quality to be described as is possible. 

2. Selecting from this collection a limited number of sam- 
ples (10 to 20) which represent the entire range of quality, 
aad which differ from each other by approximately equal 
increments of quality. 

3. Determining the quantitative description of the qual- 
ity of each sample with reference to an established zero 
point, and arranging the samples in the form of a scale. 

1. Collecting sample performances. In collecting sample 
performances for the purpose of constructing a quality 
scale, it is necessary that all of the performances be of the 
same type. For example, if samples of hand-sewing are 
being collected, all pupils should attempt to do the same 
kind of stitches with the same material and equipment. If 
samples of handwriting are being collected, all pupils should 
write the same sentences under the same conditions. There 
should be some performances approximating zero in quality, 
and there should be others which exhibit the highest possible 
degrees of excellence. All degrees of quality between these 
two extremes should be represented. In collecting samples 
of certain types of hand-sewing from which to construct a 
scale for the measurement of the quality of hand-sewing, 
Dr. Murdoch secured samples not only from school children, 
but from some adults who were generally recognized as being 
expert needlewomen, and also from the inmates of an insti- 
tution for the feeble-minded. This procedure resulted in a 
wide range of quality. The number of samples collected is 
unimportant, except that this number should be sufficiently 
large so that it is reasonably certain that all degrees of qual- 
ity will be represented. Most recent makers of scales have 
started with at least a thousand samples in their original 
collections. 
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2. Selecting samples to form a scale. In selecting from all 
of those collected a set of samples to form a scale, the ob- 
jective to be realized is a small number of samples (10 to 20) 
which will be representative of the entire range of quality 
from the poorest sample collected up to the best sample 
collected. It is desirable that the differences in quality 
between successive samples be approximately equal. Any 
plan of selection which will accomplish this purpose is justi- 
fied. The best procedure is to make a series of selections. 
For example, one may start with 2500 samples of hand- 
writing. These samples should be numbered for purposes 
of identification. The first selection may be made by a sin- 
gle judge who sorts the 2500 samples into 10 piles, represent- 
ing 10 different degrees of quality of handwriting. Any 
sample which is incomplete or otherwise unsatisfactory for 
scale purposes should be discarded. After the first sorting 
is completed the judge should examine each pile, and make 
any changes which seem desirable. Then he should select 
25 or 30 samples from each pile which will be representative 
of the pile. This will reduce the number of samples to 
nearly 300, but they should be representative of the origi- 
nal collection. 

From this derived collection a selection of 50 or 60 samples 
should now be made. In doing this the independent judg- 
ment of a number of competent judges should be secured. 
Each judge should be asked to sort the 300 samples into 10 
piles. They should be instructed to make the differences in 
quality between these successive piles approximately equal. 
The piles should then be numbered 1, 2, 3, 4, . . . . 9, 10, be- 
ginning with the pile representing the lowest degree of qual- 
ity. A record sheet similar to that illustrated in Table III 
will be needed for recording the results of the sorting by 
each judge. After each sorting the samples should be thor- 
oughly shuffled, so that no judge will be influenced by the 
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sorting of any other. The totals represent the consensus of 
opinion of these judges with reference to the quality of the 
respective samples. The sample which has the smallest 
total is the one which is judged to possess the least quality. 


TABLE III. RECORD SHEET FOR RECORDING JUDGMENTS 
OF JUDGES CONCERNING THE QUALITY OF SAMPLES 


Judge 

Number 

of OO 
sample | 4 |'p | c | D | E Total 
1 8 tf 5 9 is 36 
2 3 2 1 uu 2 9 
3 4 6 4 5 4 23 
4 He 5 8 Tf 6 33 
5 10 9 10 8 10 AT 
6 

q 

8 

9 

10 


The sample which has the largest total is judged to repre- 
sent the highest degree of quality. It is wise to secure the 
judgments of 10 to 30 judges at this stage of the selection. 

By referring to the totals, 50 to 60 samples should now be 
selected, which are representative of the entire range of 
quality. These samples should then be submitted to a still 
larger number of judges and the procedure described above 
repeated. 

Using the totals for the 50 to 60 samples as a basis, it 
should now be possible to select the samples intended for 
use in the final scale. Usually, the selection will be made 
by taking the sample having the smallest total and the sam- 
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ple having the largest total, and those samples which appear 
to represent equal increments of quality between these two 
extremes. The number chosen at this time should be not 
less than 10, nor more than 20. Occasionally, it may be 
desirable to have more than 20 samples. It is possible that 
a few samples chosen at this time may later prove unsatis- 
factory and it will be necessary to reject them. It is, there- 
fore, wise to select a few more samples than are to be in- 
cluded in the final scale. 

The process of selection by successive approximations 
can be greatly extended when desired. By increasing the 
number of judges used at each stage, and by increasing the 
number of successive selections, the process of making a 
quality scale can be greatly elaborated. Scale-makers ap- 
pear to have secured satisfactory results by making three or 
four successive selections. It is, however, necessary that 
competent judges be used at each stage of the work. In 
our illustration, the first selection was made by one judge. 
Some test-makers have used from five to ten judges for this 
first step. This greatly increases the labor and does not 
appear to be necessary. For the final selection the judg- 
ment of forty to fifty competent persons should be sought. 

The test of the authenticity of the selection is in the de- 
termination of the numerical values which represent the 
degrees of quality of the samples chosen. If these numeri- 
cal values are such that the samples differ from each other 
by approximately equal amounts of quality a good selection 
has been made. If the differences are not approximately 
equal the selection was not a good one. 

3. Determining the quantitative description of the quality 
of the samples. Three methods have been used for deter- 
mining the quantitative descriptions of the samples which 
have been chosen for a quality scale. Two of these methods, 
which may be called subjective, were described by Thorn- 
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dike in connection with the derivation of his handwriting 
scale, which was the pioneer measuring instrument of this 
type. The first method is to ask competent judges to assign 
numerical values to the samples, such as 1, 2, 2.7, 3, 3.4, 4.0, 
etc. The average of the values assigned to a sample by a 
large number of competent judges is then taken as the nu- 
merical description of the quality of the sample. When a 
value of 1 is assigned to the sample it is judged to possess the 
least quality. It is assumed that this quality is just one unit 
above zero. This assumption may or may not be justified 
in a particular case. When it is not, the values determined 
by this method will need to be shifted upward or downward, 
depending upon the location of the zero that is finally agreed 
upon. The method possesses another limitation in that 
competent judges are unable to discriminate between very 
poor samples to the same extent that they are able to 
discriminate between samples possessing moderate quality. 
The same is true of very good samples. This limitation, of 
course, could be removed if we were able to include in our 
final selection a number of samples at each end of the 
scale which were later discarded. 

Modifications of the method. This method may be mod- 
ified when revising an existing scale or when constructing a 
scale in a field where there is already a scale.! Instead of 
securing estimated values, the samples may be rated on the 
existing scale by a large number of competent judges. The 
average or median of their ratings would be taken as the 
scale values of the samples. 

The second method requires that the samples be ranked 
in order of quality by each judge. This ranking should be 
described with reference to some arbitrary ranking which 
will approximate the true ranking. Probably the best rank- 


1 Trabue, M.R. “Supplementing the Hillegas Scale”; in Teachers Col- 
lege Record, vol. xvi, pp. 51-84 (January, 1917). 
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ing for this purpose is the one according to the totals on the 
basis of which the final selection of samples was made. In 
Table IV, which illustrates the method of recording the rank- 
ings, the samples beginning with the highest were ranked 


TABLE IV. FORM OF RECORDING THE RANKINGS OF SAM- 
PLES BY JUDGES IN ORDER TO SECURE PER CENT OF 
“BETTER JUDGMENTS” 


Judges P 
No. a Total nit 
CE ae | above 
ROARED |e TE V VI | Vil | VIII| 1X x 
oe ee | ee fe 
79 | A |B A A A B B A A A be 70 
84 BIA A B A A A B B A 6 60 
42 |AJA A B A A A B A A 8 80 
108 BIA A A B A A A A B a 70 


as shown in the first column. Sample number 79 had the 
highest total in the final selection. Sample number 84 came 
next, and so on. When arranging the samples in order of 
merit judge I ranked sample number 79 above sample num- 
ber 84. This is shown by the letter A. This judge ranked 
sample number 84 below sample number 42. This fact is 
shown bya B. After the rankings of all judges have been re- 
corded, the total number of judges who ranked each sample 
above the one just below it in the arbitrary ranking should 
be entered in the next tothe last column. In the last column 
the per cent of “better judgments” is given. This is obtained 
by dividing the numbers in the next to the last column by 
the total number of judges. From the per cent of better 
rankings the numerical description of the quality of each 
sample can be calculated. 

Table V has been constructed from data given by Dr. 
Murdoch in her description of the construction of her scale 
for measuring certain elements of hand-sewing. The first 
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TABLE V. DIFFERENCES IN QUALITY CORRESPONDING TO 
PER CENT OF “BETTER JUDGMENTS” 


(Arranged from Tables 2 and 5 of Dr. Murdoch’s monograph) 


Identification Per cent of judges - Difference in 
number of placing sample i quality in 
sample above next sample terms of P.E. 
544 42 —.299 
977 88 1,742 
652 76 1.046 
976 56 224 
900 64 532 
381 68 694 
322 68 694 
653 66 612 
709 66 .612 
821 78 1.143 
497 58 .299 
671 80 1.246 
—6ll 60 .876 
473 72 865 
678 56 224 
418 56 224 
448 82 1.355 
440 60 . 876 
489 80 1.246 
528 84 1.472 
532 84 1.472 
530 


column gives the identification numbers of the 22 samples in 
the final selection. The second column gives the per cent 
of 50 judges placing each sample’ above the next sample. 
In the third column the differences in quality between suc- 
cessive samples are given in terms of P.E. These differ- 
ences are obtained from a table similar to Table I.! If we 
look in this table for 42 per cent, we find that the P.E. 
equivalent for 42.07 per cent is .30 P.E. In Table V the 
difference in quality between sample number 544 and 977 
1See page 96. 
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is given as —.299. The procedure followed in ranking the 
samples is such that the signs of the P.E. values are just 
the opposite of those given in Table I. With this change 
the differences in quality in terms of P.E. may be read di- 
rectly from Table I. 

This method of determining the quantitative description 
of the quality of samples is based upon the theorem that 
differences in quality which are noticed equally often by 
competent judges are equal in magnitude unless they are 
noticed by all such observers, or are noticed by only 50 per 
cent of the observers. The method also assumes that the 
errors in the judgments expressed by an individual judge 
form a normal distribution about zero, and that the vari- 
ability of this distribution is the same for all individuals and 
all samples, 

Location of zero point. The location of the zero point is 
usually arbitrary. Dr. Murdoch in constructing her scale 
for measuring certain elements of hand-sewing was fortunate 
in securing a sample of hand-sewing for which she obtained 
an average value of approximate zero when the opinions 
of a number of competent judges were secured in regard 
to its quality. When a scale-maker is not fortunate in hav- 
ing a sample which the judges agree upon as representing 
zero, it is necessary to secure the consensus of their opinions 
concerning the distance of certain samples above zero. 

The subjectivity of these methods. Both of these meth- 
ods for determining numerically the descriptions of the qual- 
ities of the samples chosen for a scale are subjective, in that 
they are based upon the opinions of judges. It should, how- 
ever, be noted that competent judges are to be used in every 
case, and that the values represent the consensus of opinion 
or average estimate of these judges. The procedure may be 
justified by pointing out that this is exactly the way quality 
is defined. The quality of a painting is exactly what com- 
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petent judges consider it to be. As it is usually thought of, 
the quality of handwriting is defined in terms of the con- 
sensus of opinion of competent persons. Hence, the fact 
that the methods just described are based upon opinions 
does not constitute a valid criticism. 

Objective methods of scale construction. Some scale- 
makers, notably Ayres and Freeman, have attempted to se- 
cure some objective index of quality. In constructing his 
handwriting scale, Ayres defined legibility, which was the 
quality he considered, in terms of the average rate at which 
the samples could be read. He assumed that, the other 
factors being constant, the legibility of a sample of hand- 
writing was inversely proportional to the number of seconds 
required for reading it. He therefore had a large number 
of samples, which he collected, read by a number of trained 
persons under carefully controlled conditions. From the 
data thus obtained he calculated a numerical index of the 
legibility of each sample. He then selected a group of sam- 
ples which differed from each other by approximately equal 
amounts of legibility, and which for other reasons seemed 
suitable for being included in a scale. He assumed that 
the numerical indices of legibility for an unselected group of 
samples would form a normal distribution. The base line of 
this normal distribution from — 2.50 to + 2.50 was divided 
into one hundred parts.!_ Samples were then picked out to 
represent the numerical values of 20, 30, 40, ete. 

Freeman took such qualities as uniformity of slant, uni- 
formity of alignment, and so forth, and attempted to secure 
objective descriptions of these characteristics of handwrit- 
ing. He used these objective descriptions in much the same 
way that Ayres used his data in constructing his scale. 


1 Ayres’s account of this scale does not state definitely that the extent of 


the base line was taken as 5 ¢. However, there is evidence which points 
to this choice, 
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Using a quality scale. The description of performances 
by means of a quality scale is subjective; i.e., it depends 
upon the person giving it. Hence, it is necessary to exercise 
care in order to reduce the error of measurement due to sub- 
Jectivity toa minimum. This is accomplished by following 
a systematic procedure by training. Most scales are accom- 
panied by detailed directions for their use. These should be 
followed. Two or three hours of practice will also reduce 
the degree of subjectivity. 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1, What is a scale of quality? 

2. Quality scales are needed in the description of what types of perform- 
ances? R 

8. What are the essential steps in the construction of a quality scale? 

4. What criteria should govern one in a collection of samples for the 
construction of a quality scale? 

5. What are the two subjective methods for selecting and evaluating 
samples for a quality scale? 

6. What assumptions are involved in each method? 

7. What are the sources of difficulty in securing accurate measurements 
of quality by means of quality scales? 

8. How did Ayres construct his scale of quality in handwriting? Is his 
method preferable to the one used by Thorndike? Give reasons. 

9. Which is the best general handwriting scale? Look up the experimen- 
tal evidence. 
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CHAPTER VII 
DERIVED SCORES 


The lack of a common unit of measurement. A variety of 
units are used in describing the performances of pupils 
according to the methods considered in the two preceding 
chapters. The unit of work, in terms of which the rate of a 
pupil’s performance is described, is generally determined by 
the structure of the test. Consequently, different tests are 
likely to define different units of work. Even in the case of 
tests where the units are called by the same name, they are 
likely not to be equal in magnitude. For example, in the 
Courtis Standard Research Tests, Series B, an example is 
taken as a unit of work; but an example in the addition test is 
not equal to an example in the subtraction test. Each of the 
four tests in this series has a different unit, although they are 
called by the same name. Accuracy is generally measured 
on a scale which ranges from zero accuracy to perfect accu- 
racy. The unit is one hundredth of this range. However, 
both zero accuracy and perfect accuracy depend upon the 
standard of accuracy which is adopted for a given test. Since 
standards of accuracy vary for different tests, it follows. that 
the unit of accuracy also varies. Hence, in the description of 
both rate and accuracy, different units of measurement are 
likely to be employed with different tests. 

The unit used in describing a level of difficulty which a 
pupil or class reaches on a scaled test is a measure of the 
variability of the ability of a certain group of pupils. The 
median deviation of the variability (P.E.) is one measure 
which has been used asa unit by a number of test-makers, 
such as Woody, Trabue, Hotz, and Henmon. Other work- 
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ers have used the standard deviation (a) or some fraction 
of it as the unit. The constancy of a measure of variability 
depends upon the definition of the group of pupils whose 
abilities are considered. Grade groups have been most 
frequently used, but a given grade group, such as the fifth 
grade, is not well defined. The practice with reference to 
the classification and promotion of pupils is not uniform. In 
some school systems a distinct effort is made to classify the 
pupils so that the variability with respect to the abilities 
engendered within the class is reduced to a minimum. In 
other school systems pupils are classified on a different basis 
and, as a result, the variability of these belonging to a given 
school grade in such a school system is much greater. Thus 
two test-makers utilizing different school systems in the 
construction of similar tests might define units of difficulty 
differing in magnitude. Furthermore, the variability ap- 
pears to increase slightly as we advance from a lower to a 
higher grade. If this is the case, the size of the unit would . 
depend upon the school grade that is chosen to define it. 
The median deviation (P.E.) of the third grade would not be 
equal to the P.E. of the eighth grade. Hence, a measure of 
the variability of grade groups does not furnish us with a 
constant unit for describing the performances of pupils. 
The lack of a common zero point. Not only are different 
units used in the description of the performances of pupils, 
but the descriptions of quality and level of difficulty reached 
are likely to be expressed with reference to different zero 
points. The standard of accuracy adopted establishes the 
zero point, and, as we have already shown, the standards used 
are arbitrary. Hence, the measures are expressed from arbi- 
trary zero points and do not necessarily mean no accuracy. 
The zero points of quality scales are more uniform in mean- 
ing. In the case of scaled tests, the makers have generally 
attempted to establish zero points which approximate ab- 
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solute zero, and which might be interpreted as meaning the 
absence of ability or not any of the ability being measured. 
It is, however, not certain that they have succeeded in their 
attempts, and therefore it is likely that there is little uni- 
formity in the zero points which have been established for 
scaled tests. 

In the case of the rate of work, zero means not any rate, 
or that the pupil failed to do a single unit of work within the 
time allowed. However, zero rate cannot be interpreted as 
meaning the possession of zero ability. A pupil may possess 
some ability in a given field, such as silent reading, spelling, 
or the operations of arithmetic, and yet be unable to do a 
single unit of work within the time allowed because the ex- 
ercises presented to him are more difficult than he can do. 
For this reason, if measures of rate of work are thought of in 
terms of ability rather than merely as the pupils’ rates of 
work, we have these measures also expressed with reference 
to different zero points. 

The need for a common unit and a common zero point. 
Since no two tests employ the same unit or the same zero 
point, except by chance, the scores used to describe the per- 
formances of pupils are not comparable. A score of 19 on 
one test does not, except by chance, mean the same thing as 
the score of 19 on another test. Considerable inconvenience 
results from this condition. Such point scores are without 
meaning until they are compared with norms. Since differ- 
ent units and different zero points are used, this makes nec- 
essary a different set of norms for each test, even in the case 
of tests designed to measure the same abilities. This adds 
to the labor of test construction, and makes it necessary 
for the test user to have at hand a statement of the norms 
for all tests with which he is dealing. It is not possible to 
combine the scores obtained from two or more tests, or to 
compare them in any precise way, without_ employing an 
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elaborate statistical procedure to reduce them to the same 
scale. The recording of scores on permanent record cards 
is also complicated. It is necessary to make a record not 
only of the scores but also of the norms. Otherwise, with- 
out first looking up the norms, there can be no interpreta- 
tion or comparison of the scores which pupils make at suc- 
cessive periods of testing. 

Derived scores in terms of a common unit and a common 
zero point. The description of a pupil’s performance ac- 
cording to the rules which accompany a test may be called a 
point score. In order to secure a common unit and a com- 
mon zero point a number of plans have been proposed for 
translating the point scores into derived scores. We shall 
consider certain of these proposals. 

The variability of chronological-age groups as a common 
unit. As indicated above, a measure of the variability of 
grade groups does not furnish us with a constant unit, be- 
cause the definition of such groups is not fixed. In order to 
avoid this limitation, McCall has proposed that we use the 
group of pupils whose chronological ages fall between twelve 
and thirteen years. Since practically all pupils of these ages 
are to be found in our public schools, twelve-year-old pupils 
would always be selected no matter from what school sys- 
tem they were taken. Such a group would always possess 
the same average general intelligence, provided it was a ran- 
dom selection from a typical population. 

The character of the instruction which the pupils had re- 
ceived would doubtless affect their variability slightly. It 
has been pointed out that the usual plan of instruction 
does not furnish the bright pupils with adequate opportuni- 
ties for learning. The emphasis has been placed upon the 
instruction of backward and dull pupils. If our procedure 
were modified so that backward and dull pupils received less 
attention, and the bright pupils received more, it might well 
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be that the variability of the chronological-age groups would 
be affected. Furthermore, we do not know that the variabil- 
ity is the same for different abilities. For example, the vari- 
ability might be greater for the ability to add than for the 
ability to subtract. The variability of the ability to read 
might differ from that of ability in the field of music. 

Another objection to the universal application of this pro- 
posed unit is that some abilities have not been acquired by 
all twelve-year-old pupils. This would be true of many abili- 
ties engendered in the high school, because most twelve- 
year-old children have not yet entered high school. In the 
case of tests designed for use in the primary grades it would 
be necessary to introduce some modifications when apply- 
ing them to twelve-year-old pupils, most of whom would be 
found in the intermediate and grammar grades. In many 
cases the measuring instruments would not be suitable for 
use with most twelve-year-old pupils and, consequently, 
would fail to yield a measure of their abilities. It has been 
suggested that this limitation might be overcome by using 
sixteen-year-old pupils for abilities engendered in the high 
school, and eight-year-old pupils for measuring instruments 
designed for use in the primary grades. This suggestion 
appears to be feasible in the case of some tests designed for 
use in the primary grades, but for abilities engendered in the 
high school a difficulty is introduced by reason of the fact 
that there is a differentiation of the students with respect to 
courses studied. All pupils do not study the same school 
subjects, and there is evidence to show that we do not secure 
a random sampling of pupils in the different school subjects. 
For example, one investigation indicates that pupils study- 
ing Latin possess a higher average general intelligence than 
do pupils studying commercial subjects. 

However, in spite of these limitations, the proposal to use 
the variability of the ability of twelve-year-old pupils will 


150 THEORY OF EDUCATIONAL MEASUREMENTS 


result in a unit which is approximately the same for a num- 
ber of tests. Whenever the conditions are such that it can 
be used, a grouping of pupils on this basis is to be preferred 
to a grouping on the basis of school grade. 

A proposed common zero point. A common zero point is 

also necessary. It is generally difficult or impossible to de- 
- termine an absolute zero point which will mean not any of 
the ability in question. Therefore, in order to secure uni- 
formity, McCall has proposed the use of a zero point 50 be- 
low the average ability of twelve-year-old pupils. He shows 
that this zero point approximates the determinations of ab- 
solute zero points made for a number of particular tests. 
Even if such a zero point does not represent absolute zero of 
ability, it will be useful because it will always have a mean- 
ing of being just 50° below the average ability of twelve-year- 
old pupils. 

McCall’s derived scores. McCall! has used a unit .1o of 
the distribution of the abilities of all twelve-year-old pupils. 
This unit he calls “T,” after Thorndike and Terman. His 
scale extends from 50 below the average of the ability of 
twelve-year-old pupils up to a point 50 above this average. 
Thus, he has a scale of 100 T units. A pupil’s performance 
is first described in terms of a point score. In the case of 
the Thorndike-McCall Reading Scale, in connection with 
which McCall described his plan for derived scores, this 
point score is the number of questions answered correctly. 
The T score which is equivalent to each point score is deter- 
mined by ascertaining, for the group of twelve-year-old pu- 
pils, the per cent of pupils exceeding each point score.2. The 

1 McCall, W. A. “A Proposed Uniform Method of Scale Construction”; 
in Teachers College Record, vol. xxu, pp. 31-51 (January, 1921). 

? More exactly, this is the per cent exceeding plus half of those reaching 
each point score. The number of pupils making a given point score is di- 


vided, half being grouped with those making higher scores and half with 
those making lower scores. 
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relation between the per cent of pupils exceeding given point 
scores and the corresponding T scores is given by Table I. 
For example, McCall found, in the case of the Thorndike- 
McCall Reading Scale, that 93.1 per cent of the twelve- 
year-old pupils exceeded a point score of 12. By referring 
to Table I, we find that 93.1 per cent of the pupils in a nor- 
mal distribution, which is assumed for the twelve-year-old 
group, are above a point 1.500 below the average of this 
group. If the position of this point is expressed with refer- 
ence to a zero point taken 50° below the average it would be 
+ 3.50. Since T, the unit used by McCall, is .1o, the posi- 
tion of this point would be described by the score 357. 
Therefore, a point score of 12 on this test is to be trans- 
lated into a derived score of 35T on McCall’s uniform scale. 

This plan of calculating derived scores may be applied to 
all tests which are appropriate for twelve-year-old pupils. It 
is only necessary to ascertain for each point score which the 
test yields the per cent of twelve-year-old pupils who exceed 
it. By means of Table I we will be able to ascertain the T 
score which is equivalent to any given point score. The 
adoption of this plan would mean that all scores would be 
expressed in terms of a common unit,? and with reference to 
a common zero point. This would result in the scores 
yielded by different tests having a similar meaning. For 
example, a score of 35 would always represent that degree of 
ability which was exceeded by 93.1 per cent of all twelve- 
year-old pupils. Similarly, a score of 62 on this scale would 
represent that degree of ability which is exceeded by only 
10.6 per cent of all twelve-year-old pupils. 

The advantage of using such derived scores is obvious. 
Since all scores are expressed in terms of the same unit, and 


1See page 96. ett 
2 In the case of tests in different fields this assumes that the variability 
of the distributions of ability is the same. 
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with reference to the same zero, they may be added or sub- 
tracted. Only simple arithmetic is required to find a pupil’s 
average score. A score ofa given magnitude will always have 
the same meaning with reference to the norm for twelve-year- 
old pupils. The norm (average) for this group will always 
be 50. However, the necessity for separate sets of norms 
for different abilities, in the case of groups other than twelve- 
year-old pupils, will not be eliminated. Thus we should ex- 
pect to have different sets of norms for different tests and 
to some extent for different plans of school organization 
and for different courses of study. 

Pintner’s derived scores. Pintner! has proposed a modi- 
fication of McCall’s procedure. Instead of expressing all 
scores in terms of the variability of one age group, he has 
prepared a separate scale for each age group. This plan has 
the advantage of assigning toeach pupil a score which directly 
compares his ability with that of his own age group, but it 
does not facilitate the comparison between the abilities of 
pupils of different ages. Different units and different zero 
points are used for pupils of different ages. A common unit 
and a common zero point will be secured only for pupils be- 
longing to the same chronological age group. The plan is 
obviously more complex than the one proposed by McCall 
and has less to commend it. 

Derived measures expressed in terms of grade norms. It 
has been proposed that the point scores yielded by a given 
test be expressed in terms of the grade norms for that test.” 
For example, one set of proposed grade norms for the 
Courtis Standard Research Tests, Series B, in terms of the 

1 Pintner, Rudolph, and Marshall, Helen. ‘‘A Combined Mental- 


Educational Survey’; in Journal of Educational Psychology, vol. x11, p. 32 
(January, 1921). 

? Buckingham, B. R. “Suggestions for Procedures Following a Testing 
Program: I, Reclassification”; in Journal of Educational Research, vol. u, 
p. 787 (December, 1920). 
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number of examples attempted by pupils just entering the 
respective grades are given in the table below: 


Grade Addition Subtraction | Multiplication| Division 
VIII 11.2 12.2 10.9 10.1 
VII 10.3 Ae 9.6 8.9 
VI 9.2 9.8 8.7 7.1 
V 8.0 8.2 Chie be 5.3 
IV 6.2 6.2 3 Bs 


If a pupil attempts 9 examples in addition he is shown to 
have a degree of ability between the norm for the fifth grade 
and the norm for the sixth grade. The norms, as given in 
the above table, are in terms of the abilities of pupils who 
are just entering the respective grades. Therefore, this pupil 
may be said to have a degree of ability ten twelfths greater 
than that possessed by the average of the pupils just begin- 
ning the fifth grade. According to this plan, this pupil’s 
ability in addition could be represented by a derived score of 
of 5.8. 

This proposal for translating point scores into derived 
measures possesses very obvious limitations. In the first 
place, grade norms are not constant. They depend upon 
the organization of the school and the effectiveness of the in- 
struction. If a measure of the variability of grade groups is 
unsatisfactory as a unit, certainly a scale consisting of the 
grade norms will be even more unsatisfactory. However, the 
most serious limitation is that the plan does not provide any 
means for translating either low scores or high scores into 
derived measures. Such a scale cannot extend below the 
norm for the lowest grade to which the test is given. It 
cannot extend above the norm for the highest grade to 
which it is given. Therefore, we have a scale which is 
inadequate at both ends. Furthermore, the plan presents 
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difficulties of application. For these reasons, derived meas- 
ures in terms of grade norms should not be used. They have 
nothing to commend them except in the grade placement 
of pupils. Even in this activity they possess no essential 
advantage over other derived measures. 

Derived measures expressed in terms of grade percentile 
scores. A pupil’s performance may be described in terms 
of the position which it occupies in the standard distribu- 
tion for his grade. This is accomplished by dividing the 
distribution so that each division contains the same per 
cent of the total number of scores. One plan is to make 
four divisions, or quartiles. A pupil’s performance may 
then be described as being in a given quartile, such as the 
second quartile, or the fourth quartile. Other plans require 
a division of the distribution into 10, 20, or 100 parts. When 
it is divided into 100 parts each part is called a percentile, 
and a pupil’s ability might be described as that of the 80 
percentile or the 57 percentile. The objections which have 
previously been stated with reference to the instability of 
grade distributions apply to this proposal. There are also 
additional reasons why this plan is unsatisfactory. While 
the derived scores are expressed in terms that can be easily 
understood, they do not possess the qualities which are pos- 
sessed by derived scores expressed in terms of the variability 
of the chronological-age group. The unit is not constant. 
In calculating percentile scores the distribution is divided so 
that equal areas are obtained. For this reason the divisions 
do not mark equal distances on the base line of the distribu- 
tion. The result of this is that the difference between the 
degrees of ability represented by a 40 percentile score and 
a 45 percentile score is not at all equal to the difference 
between the degrees of ability represented by a 90 per- 
centile score and a 95 percentile score. The latter is much 
larger. 
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Measures expressed as achievement ages. In using the 
Binet Scale for Measuring the Intelligence of Children, a 
pupil’s performance is first described in terms of his success 
or failure in doing certain exercises. This description is then 
translated into a derived measure, called mental age. A 
pupil is said to have a mental age of nine years when the 
point score which describes his performance is equivalent to 
that of the average point score of nine-year-old children. 
Likewise, a pupil is said to have a mental age of twelve 
years when the point score that describes his performance 
is equivalent to the average point score of all twelve-year- 
old children. This plan of translating point scores into cor- 
responding age scores has been applied to other intelligence 
tests. The procedure for obtaining the basis for making this 
translation has been to give the test to an unselected 
group of pupils for each age, and in this way determine the 
average point score which is made by the pupils of each age 
group. These chronological age norms then become the basis 
for translating point scores into the corresponding mental- 
age scores. The age group whose norm is equivalent to a 
pupil’s point score becomes his mental age. 

This procedure has been applied to achievement tests. 
McCall, Pintner, and others have determined the norms for 
chronological-age groups and used these as a basis for trans- - 
lating point scores into age scores. Since a chronological-age 
norm represents the average achievement of pupils of that 
age, it is necessary to introduce some modifications of this 
plan in order to provide for translating very low scores 
and very high scores into corresponding achievement ages. 
Pintner fails to make this provision. For example, the aver- 
age of the highest group in Pintner’s table is 102. The max- 
imum point score is 141. McCall has assumed a rectilinear 
relationship between point scores and age norms. On this 
basis he has extended his age scale upward and downward 
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to provide for age scores corresponding to all pot scores 
which his test yields. 

The writer ! has proposed, in the case of arithmetic and 
silent reading, that mental-age groups be used instead of 
chronological-age groups in determining the basis for trans- 
lating point scores into achievement ages. The reason for 
making this proposal is that mental age is a better criterion 
of ability in these two fields than chronological age.?, Pupils 
having the same mental age are said to possess the same 
capacity to learn. However, since the average mental age 
of an unselected chronological-age group is numerically the 
same as the average chronological age, this difference in pro- 
cedure will not produce a numerically different result. The 
use of mental-age groups has the advantage of certain logi- 

cal considerations in connection with the interpretation of 
the resulting derived measures. 

Quotients as derived measures. A score, even a derived 
measure of the kind we have described above, tells only the 
magnitude of a pupil’s achievement. This has meaning only 
when compared with appropriate norms. In most uses of 
educational measurements we are concerned more with the 
comparison of the measure of a pupil’s ability with an 
appropriate norm than with the absolute amount of his 
ability. For this reason, another type of derived measures 
have been proposed which would be an index of the rela- 
tion of the measure of a pupil’s ability to the appropriate 
norm. Mental ages have been compared with chronologi- 
cal ages by dividing the mental age of a pupil by his chrono- 
logical age. The resulting quotient is called the intelligence 


1 Monroe, Walter S. The Illinois Examination, Bulletin No. 6, Bureau 
of Educational Research, University of linois Bulletin, vol. xx, No. 9. 

2 Tt is likely that this statement is true for most other fields of ability. 

3 More strictly speaking, a pupil’s mental age is divided by the norm for 
his chronological age. 
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quotient, or .Q. McCall! and Franzen? have divided the 
pupil’s achievement age by his chronological age and called 
the result the pupil’s educational quotient (E.Q.). This 
quotient indicates the relation of a pupil’s achievement to 
his chronological age, or, more strictly speaking, to the 
norm for his chronological age. If a pupil’s achievement age 
is just equal to the norm for his chronological age his E.Q. 
will be 100. If his achievement age is above the norm for his 
chronological age his E.Q. will be greater than 100. If it is 
below it will be less than 100. The magnitude of a pupil’s 
E.Q. will, therefore, define the position of his achievement 
with reference to the norm for his chronological age. 

The comparison of a pupil’s achievement with the norm 
for his chronological age is not in itself very valuable for 
most purposes. It does not tell us whether the pupil’s 
achievement is what it should be, because we know that pu- 
pils of the same chronological age differ widely with respect 
_ to capacity to learn. In order to secure a comparison of a 
pupil’s achievement with his capacity to learn, as defined by 
his mental age, Franzen has proposed that a pupil’s E.Q. be 
divided by his IQ. ~ ] 


i. i 
LETTS CIT VE: 
1.Q jig sh eek «8 Nae oli ete Te 

C.A. 


This, however, is essentially the quotient of a pupil’s achieve- 
ment age divided by his mental age. Thus, a pupil’s achieve- 
ment is compared with his capacity to learn, or, more strictly 
speaking, with the norm for pupils possessing his capacity 


1McCall, W. A. “A Proposed Uniform Method of Scale Construc- 
tion”; in Teachers College Record, vol. xxu, p. 31 (January, 1921). 

2 Franzen, Raymond. “The Accomplishment Quotient”; in Teachers 
College Record, vol. xx1, p. 432 (November, 1920). 
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to learn. Franzen has called this quotient the accomplish- 
ment quotient (A.Q.). 

The writer! has proposed, for silent reading and arith- 
metic, that a pupil’s achievement age be divided by his men- 
tal age, or more strictly by the norm for his mental age, and 
has called the quotient an achievement quotient (A.Q.).? 
Although the plan gives results which are numerically iden- 
tical with the accomplishment quotient secured by Franzen 
by a less direct method, there is one significant logical differ- 
ence. The norms used by the writer in securing achieve- 
ment ages were mental-age norms and not chronological-age 
norms. Since this translation into achievement age was 
made primarily to facilitate comparison with mental age, 
it logically follows that mental age rather than chronologi- 
cal age should be used as a basis for grouping the pupils in 
calculating the achievement ages. From the standpoint of 
the construction of a test Franzen’s procedure is more sim- 
ple, because pupils can be grouped more easily on the basis 
of chronological age. In order to secure a mental-age group- 
ing it is necessary to give a general intelligence test as well as 
the achievement test. Since Franzen’s method gives the 
same numerical results, test-makers will probably find it 
preferable. 

The significance of the achievement quotient (A.Q.). 
The educational quotient (E.Q.) does not appear to possess 
a great deal of significance as an index of the pupil’s effective- 
ness as a learner. The achievement quotient, on the other 
hand, is an index of the ratio of the pupil’s achievement to 
his capacity to achieve. The value of the achievement quo- 

1 Monroe, Walter S. The Illinois Examination, Bulletin No. 6, Bureau 
of Educational Research, University of Illinois Bulletin, vol. xx, No. 9. 

2 A.Q. is an abbreviation for both “accomplishment quotient” and 
“achievement quotient.” Both of these terms have been given essentially 


the same meaning but the latter appears to be the preferable term. Its 
use is recommended. 
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tient will be illustrated in Chapter X, but we may indicate 
here a general basis for its interpretation. Intelligence quo- 
tients (I.Q.’s) are interpreted with reference to fixed norms 
for pupils of all ages. Pupils who have I.Q.’s of approxi- 
mately 100 are considered normal or average in intelligence. 
A pupil who has an I.Q. below 75 to 80 is described as dull, 
while one who has an I.Q. above 115 to 120 is considered 
bright. Since the median A.Q. tends to be the same for 
successive grades, we may proceed on a similar basis to set up 
levels of interpretation. For Monroe’s Standardized Silent 
Reading Test, Revised, and Monroe’s General Survey Scale 
in Arithmetic, the following basis for interpreting A.Q.’s has 
been worked out: 


Quality of pupil’s Achievement Per cent of pupils 
achievement Quotient included 
ELV SUPCTIODS 12-649 214.19 sc0u0. 0:0 } et 6: 
SE DETION A ers ie cieraisa- eine w'elvia is < 117-134 13 
IAVETAPOICE fontes ciclo aicisis se « 83-116 60 
OOD ere iels ale ee sins iw sao 71-82 13 
i 55-70 6 
A AAILO eee cic joistersisiaia olaiale' vias i Below 55 k 1 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1. In what ways are point scores unsatisfactory as descriptions of pu- 
pils’ performances? 

. What is a derived measure? 

. Explain McCall’s proposed procedure for securing derived measures 
in terms of scores. 

. What is the “educational quotient”? 

. What is the “achievement quotient”? Compare the achievement 
quotient proposed by Monroe with the accomplishment quotient 
proposed by Franzen. 

. Why are percentile scores unsatisfactory? 

. What are the objections to using grade norms as a basis for calculat- 
ing derived scores? 

8. Can the achievement quotient be extended to all school subjects? 


o9 © 
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CHAPTER VIII 


NORMS FOR EDUCATIONAL TESTS AND THEIR 
USES 


Definition of norms. As usually calculated, a norm is the 
median or average of the present attainments of a given 
group; for example, the grade norms established for Mon- 
roe’s Standardized Silent Reading Tests are the medians of 
the scores obtained by giving the test to several thousand 
pupils in each grade. The norms obtained in this way are 
used in interpreting the scores of individual pupils, and the 
median or average scores of classes and other groups of pu- 
pils. ‘These norms are commonly thought of as defining 
the degree of ability which a pupil or a group of pupils should 
possess. In some instances the idea of perfection is associ- 
ated with a norm. This method of standardizing a test 
makes it clear that we are not justified in thinking of norms 
as a statement of what ought to be. As they are now de- 
rived, our norms must be interpreted merely as representing 
the average achievement of groups of pupils under present 
school conditions. 

Types of norms. For the purpose of deriving norms, pu- 
pils may be grouped in a number of different ways. The 
most common plan is to group them according to the school 
grade to which they belong. The resulting norms are called 
grade norms. The pupils may also be grouped according to 
their chronological ages. When this is done, we secure 
chronological-age norms. When pupils are grouped accord- 
ing to their mental ages we secure menial-age norms. We 
may also determine norms based upon success or failure in 
the pursuit of school subjects or vocations. In deriving 
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success norms, pupils, or adults in the case of vocations, are 
grouped according to success or failure. The uses of these 
types of norms will be discussed later in this chapter. Cer- 
tain general limitations may be mentioned here. 

General limitations of norms. The grade placement of 
pupils is not uniform. Some school systems have only seven 
grades below high school; others have eight grades; and a 
few, nine grades. Furthermore, there are marked differ- 
ences in the promotion rates in different school systems. 
Consequently, the general plan of grouping pupils according 
to school grade is variable and depends upon the organiza- 
tion of a school system. This is one reason why grade norms 
are not satisfactory. Chronological age furnishes the most 
precise basis for grouping pupils. It is, however, incon- 
venient to effect such a grouping in standardizing a test 
because the pupils of a given age are to be found in a num- 
ber of different school grades. In the case of ages above 
fourteen, all the children are not to be found in school. A 
grouping on the basis of mental age requires that all pupils 
be given a general intelligence test in order to ascertain their 
mental ages. This greatly adds to the labor of standard- 
izing a test, and there is the added inconvenience that the 
pupils belonging to any mental-age group must be sought in 
a number of different grades. The merits of each type of 
norm, and their uses, will be considered later in this chapter. 

The basis of satisfactory norms. As pointed out above, 
the norms which we now have are merely statements of the 
average of present conditions. In order that a norm may 
represent the degree of achievement which a pupil should 
possess, certain criteria must be satisfied. These may be 
stated under three headings. 

1. Reasonable expenditure of time and effort. Investi- 
gation has revealed that when subjected to intensive train- 
ing typical pupils can attain very high degrees of ability. 
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However, after a certain level is reached, the law of dimin- 
ishing returns begins to operate, and, unless the ability en- 
gendered is very important, the expenditure of a large 
amount of time and effort cannot be justified on the basis 
of accepted educational objectives. Therefore, satisfactory 
norms must be below the maximum of possible achieve- 
ment. The achievement which is possible for a pupil de- 
pends upon his capacity to learn, or general intelligence. 
What is a reasonable achievement for one pupil may not be 
reasonable for another pupil belonging to the same school 
grade. Norms which represent the average of present 
achievements of pupils are certain not to be too high from 
the standpoint of being capable of attainment by the “‘aver- 
age”’ pupil with the expenditure of a reasonable amount of 
time and effort. Our present plan of arriving at norms, 
therefore, possesses the merit of preventing unreasonable 
ones for “average” pupils, but it must be supplemented by 
a recognition of mental age in order to obtain norms for in- 
dividual pupils. 

Investigation has revealed that in many instances the 
present achievements of pupils are far below the achieve- 
ments that can be secured by the expenditure of a reasonable 
amount of time and effort. Therefore, it is not at all certain 
that the average of present achievements furnishes us with 
norms that are sufficiently high. It may be that when we 
learn to adjust our methods and devices of instruction to the 
capacities of pupils, higher achievements may be obtained 
with even less expenditure of time and teaching effort. 

On the other hand, it is possible that the average of pres- 
ent achievements gives us norms that are too high. This is 
likely to be the case if our present courses of study provide 
for greater teaching emphasis upon certain topics than can 
be justified on the basis of future requirements for the func- 
tioning of the ability. Thus the average of present achieve- 
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ments cannot be accepted as a satisfactory norm unless we 
can show that our course of study is satisfactory. The 
standardization of educational tests is thus intimately con- 
nected with the determination of minimum essentials and 
curriculum making. 

Within the limits of reasonableness the exact magnitude 
of norms is determined by the requirements of adult activities 
and by the requirements of future school activities. A norm 
must not be thought of as representing the maximum of 
possible attainments of pupils; in fact, the most defensible 
norm may be materially below the maximum. 

2. Requirements of adult activities. The function of the 
school is to equip pupils for effective participation in the 
activities of adult life. Outside of school there are a num- 
ber of demands which are common to all, or at least to a very 
large per cent of our population. These include the reading 
of newspapers, magazines and books, writing letters, express- 
ing ideas in conversation, and the solving of simple arith- 
metical problems of everyday life. In addition to these, 
there are a large number of special demands which depend 
upon one’s occupation. In establishing our norms we must 
take into account the common demands of adult activities for 
the abilities which the school engenders. Educators differ 
concerning the extent to which public schools should prepare 
for the special demands. They probably should receive some 
consideration in establishing norms, but they should not be 
a determining factor except in the establishing of norms for 
special vocational courses. There is need for analyzing the 
adult activities which are common to a large per cent of our 
population in order to ascertain the precise demands which 
they make for the abilities in reading, arithmetic, history, 
science, ete. Our norms should be somewhat greater than 
these demands in order to provide for the loss of ability due 
to infrequent use after children leave school. 
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3. Future school requirements. A satisfactory norm 
must be efficient with respect to the future instructional re- 
quirements of the school. The subjects taught in the school 
may be classified under two heads: — tool subjects and con- 
tent subjects. The tool subjects of the elementary school 
are reading, handwriting, the operations of arithmetic, spell- 
ing, and language. In the study of a content subject, such 
as problems of arithmetic, literature, geography, history, 
science, etc., the tool subjects are used. Time and effort 
are required for acquiring skills in these tool subjects. Time 
and effort are also required when these skills are used in the 
study of content subjects. If only a small degree of skill is 
acquired, the time and effort required in the study of con- 
tent subjects will be greatly increased. For example, time 
and effort are required in learning to read. Reading is a 
tool that is used in the study of history. If a pupil enters 
upon the study of history possessing only a small degree of 
skill in reading, he will find that much time and effort are re- 
quired in the study of history. On the other hand, if he is 
equipped with a high degree of skill in reading he will find 
that the study of history requires much less time. The 
work of the school cannot attain its highest efficiency unless 
the pupils are well equipped with the tools of learning which 
they have frequent occasion to use in the study of content 
subjects. 

In the seventh and eighth grades and in the high school 
our manner of cazrying on school work by the use of text- 
books and reference libraries makes very heavy demands for 
reading. It is also our custom to require much written 
work, prepared outside of the recitation period, and in some 
subjects much written work during the recitation period. 
This custom makes heavy demands for writing, spelling, and 
written expression. In arithmetic we expect pupils to learn 
to solve problems (not examples) by solving problems. In 
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fact, we require them to solve many problems, and the solv- 
ing of problems requires the performing of arithmetical 
operations. In view of the fact that the school makes enor- 
mous demands upon its pupils for the use of these tool skills, 
it is folly not to prepare them adequately for these demands. 
In case the school fails to equip the pupils to read effectively 
it means that the numerous assignments which they will be 
asked to study will not only consume an enormous amount 
of time, but will also destroy interest in the school work be- 
cause for them reading is slow and difficult and, hence, a 
disagreeable task. 

School demands often exceed those of adult life. In gen- 
eral discussions concerning what the school should accom- 
plish, and in practically all the discussions of norms for 
particular school subjects, attention has been focused upon 
the demands of life outside of school, and the demands of the 
school have been overlooked to a large extent. This is prob- 
ably due to the emphasis upon the fact that the function 
of the school is to prepare children for effective participation ° 
in adult activities. This is a most wholesome and com- 
mendable point of view, but its acceptance should not blind 
one to the fact that the demands which the activities of the 
school make for the use of the skills engendered in the teach- 
ing of the tool subjects exceed many of the demands which 
are made by adult activities that are common to a large per 
cent of our population. The average man or woman does 
not meet as pressing demands for reading as do the pupils in 
the high school. Likewise, the demands for writing, and 
probably for the other tool subjects as well, which the pupils 
meet in school are greater than they will meet outside of 
school. 

Even if the tool subjects were not practical, it would be 
necessary for the school to teach them and teach them well 
in the first six grades in order that the pupils might be effi- 


NORMS AND THEIR USES 167 


cient in doing the work of the following grades. It should, 
however, be recognized that, in the case of the content sub- 
jects, the source of the norms is primarily the demands of 
life outside of school. 

Norms stated with reference to specific testing conditions. 
A given set of norms applies only to certain testing condi- 
tions. All elements of the testing conditions which affect a 
pupil’s performance must be specified with reference to a 
set of norms, and these norms can be used to interpret 
only those scores which have been secured in accord with 
these specifications. A similar statement may be made 
with reference to the rules for scoring the performances of 
pupils. The explanation of the nature of the test and the 
directions in regard to methods of work materially affect 
performances. Hence, in using a test it is necessary to fol- 
low explicitly the directions for using it on which the ayail- 
able norms are based. 

The effect of acquaintance with a test. The acquaintance 
of pupils with the testing procedure and with the general 
character of the test affects their performances on most tests. 
Pupils who are taking a test for the first time do not earn as 
high scores as they will make on a second or third trial of 
the test when it is repeated after only a short interval, even 
though a different form of the test is used. Consequently, 
norms which are stated with reference to first-trial scores 
cannot be used in interpreting scores derived from second 
or third trials unless these scores are properly discounted. 

The effect of acquaintance with the test upon second- 
trial scores may be determined by having the test repeated — 
soon after the first trial. In a study by the author, the IIli- 
nois General Intelligence Scale showed an average practice 
effect of approximately six months of mental age when the 
returns from the eighth-grade pupils were not used. In this 
grade unusual conditions appear to have prevailed, and 
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when the scores from it were included the practice effect was 
approximately eight months. For Monroe’s General Sur- 
vey Scale in Arithmetic the average practice effect was 
found to be approximately 3.2 points in grades three to five, 
and 4.5 points in grades six to eight. Although the effect of 
practice has not been generally determined for educational 
tests, these determinations are probably not larger than 
what would be found for most of our tests. It appears that 
in many cases second-trial scores will be about 10 per cent 
higher than first-trial scores. 

The increase of third-trial scores over second-trial scores 
is less, but not negligible. If an interval of several weeks 
intervenes between two trials the increase will be less, if no 
attention has been given to the exercises of the test in the 
interim. If the exercises have been made the basis of in- 
struction the increase is likely to be greater. The recency of 
instruction on the topics covered by the test also affects a 
pupil’s performance. Norms which are stated for pupils who 
have had no recent instruction on the topics covered by a 
test will not be appropriate for interpreting the scores derived 
from pupils who have had recent instruction on that topic. 

The effect of coaching. When pupils are coached upon 
the test between trials, or are given training in doing the ex- 
ercises of the test, much greater gains are to be expected. 
The Illinois Examination was given in one school where it 
appears that the teachers decided that their pupils would 
profit by receiving instruction upon the type of exercise that 
this battery of tests contains. The teachers did not know 
that the second form was to be given and, therefore, did not 
have in mind preparing their pupils for it. Their instruc- 
tion was to a slight extent based upon the Form I test 
papers. Finding that their pupils were rather weak in 
knowledge of vocabulary and in synonym-antonym, special 
training was given along these lines in language work. In 
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arithmetic, practice was given upon those combinations in 
which the pupils seemed weak. In reading there was a 
special drill for increasing the rate of silent reading. 

The first form of the Illinois Examination was given to 
these pupils in November, and the examination was re- 
peated, using the second form, in May. Although it is com- 
monly assumed that general intelligence is unaffected by 
school instruction, the difference between the median scores 
for these two trials was equivalent to slightly more than 
four years of mental age. Obviously, a median growth of 
four years in general intelligence is impossible during a pe- 
riod of six months. Thus, we are forced to the conclusion 
that the training that the school gave these pupils upon 
the exercises of the Illinois General Intelligence Scale intro- 
duced a fictitious gain in the second-trial scores of approxi- 
mately three and one half years. 

One would naturally expect that the achievement scores 
in arithmetic and silent reading would be materially affected 
by instruction. The gains were relatively somewhat larger 
than the gains in mental age. It is probable that some of the 
increase in the May scores over the November scores was 
due to the pupils being more familiar with the testing proce- 
dure. The explicit training which they received is also a 
factor. However, some of the gain doubtless represents real 
growth in achievement, but we have no way of knowing the 
amount of this growth until we have determined the amount 
of gain due to coaching and to acquaintance with the test- 
ing procedure. 

The equivalence of the duplicate forms. Most of our 
educational tests are being constructed so that there are two 
or more duplicate forms. These consist of exercises that are 
the same in kind, but require different answers. The inten- 
tion is to secure duplicate instruments which will yield equiv- 
alent measures. Frequently this ambition is not realized, 
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Investigation has revealed that duplicate forms may not 
yield equivalent measures, even though a great deal of care 
was exercised in their construction to make them equivalent. 
For example, the three forms of Monroe’s Standardized 
Silent Reading Tests are not equivalent, and it has been 
necessary to announce correction numbers in order to make 
comparisons between scores yielded by the duplicate forms.! 
It has also been found that the duplicate forms of the Bur- 
gess Picture Supplement Scale for measuring reading ability 
do not yield equivalent measures. 

The equivalence of duplicate forms may be determined by 
arranging copies of the several forms of a test in alternate 
order. For example, if it is desired to determine the equiv- 
alence of three forms of a test, a pile of test papers should be 
prepared in which the first paper is a copy of form 1, the sec- 
ond acopy of form 2, the third a copy of form 3, the fourth a 
copy of form 1, the fifth a copy of form 2, the sixth a copy of 
form 3, and soon. If this pile of test papers is distributed to 
pupils as they are seated, each form of the test will be given 
to a random sample of the pupil population. A sufficient 
number of pupils should be tested so that two or three hun- 
dred scores will be obtained for each form of the test. A 
comparison of the average scores will indicate the amount 
of non-equivalence of the different forms of the test, and will 
also furnish a basis for calculating correction numbers that 
may be used when it is desired to compare the scores yielded 
by different forms.? 


1 Monroe, Walter S. Report of the Division of Educational Tests for 1919- 
20. Bulletin No. 5, Bureau of Educational Research, University of Illinois 
Bulletin, vol. xv, no. 21, p. 19. 

2 Monroe, Walter S. The Illinois Examination. Bulletin No. 6, Bureau 
of Educational Research; University of Illinois Bulletin, vol. xrx, No. 9. 
This bulletin gives an account of the determination of the amount of non- 
equivalence of the different forms of this battery of tests, and also the cor- 
rection numbers which are proposed for use. 
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Since duplicate forms seldom yield measures which are 
equivalent, it is obviously necessary that we have separate 
norms for each form, or that we use correction numbers to 
reduce the scores yielded by the different forms to a common 
basis. Failure to do this may introduce serious errors of 
interpretation. 

Errors of interpretation vs. errors of measurement. Er- 
rors of interpretation should be carefully distinguished from 
errors of measurement. There is no reason to believe that 
second-trial scores are not just as accurate absolute measures 
of ability as first-trial scores. In fact, they are probably 
more accurate, since the pupils are acquainted with the test 
and hence it is likely that the testings are more nearly con- 
stant for all pupils. However, a serious error of interpreta- 
tion is likely to be introduced if the scores from the two 
trials are interpreted with reference to the same norms. 
Errors of interpretation are also likely to be introduced 
when the directions for administering the test are modified 
in any way. 

Different uses of educational tests require different types 
of norms. In Chapter III we pointed out a number of ac- 
tivities of the school which require the measurement of the 
abilities of pupils. Different types of norms are used in the 
interpretation of measures of ability in these activities. We 
shall consider the types of norms which these uses of educa- 
tional measurements require. 

Promotion and classification of pupils. In grouping pu- 
pils for the purpose of instruction the norms required are 
naturally those which are based upon the grouping desired. 
For the placement of pupils in grades, grade norms are de- 
manded for both measures of intelligence and achievement. 
When the pupils belonging to a given grade are classified into 
fast, average, and slow sections on the basis of their intelli- 
gence quotients, the norms required are those based upon the 
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probability of success in the respective sections. For example, 
the norm for a fast section should represent the degree of 
brightness below which success in the fast section isnot likely. 

Educational and vocational guidance. Advice with ref- 
erence to future educational and vocational activities should 
be given largely on the basis of the probable success in pur- 
suing such activities. For this purpose, we need norms 
which define the degrees of general intelligence and achieve- 
ment required for probable success in various educational 
and vocational activities. Such norms have been obtained 
by measuring the ability of persons now pursuing these ac- 
tivities, both successfully and unsuccessfully. The result- 
ing information is then sorted on the basis of the success of 
the persons from whom it was obtained. A norm is thus 
secured below which success is not probable. 

Evaluation of school efficiency. The efficiency of a school! 
system or a subdivision of it cannot be evaluated or meas- 
ured directly by giving tests to the pupils. Measurements 
of the achievements are not in themselves a measure of the 
efficiency of the instruction and training which the school is 
giving to the pupils. It is only through the comparison of 
such measures with appropriate norms that we are able to 
make any inference concerning the efficiency of the instruc- 
tion and training. If the deviations of pupils’ scores from 
the norm are due, either directly or indirectly, to the instruc- 
tion and training these deviations then become an index or 
indirect measure of the efficiency of the school. If they are 
due to other causes, they will not be a true measure of the 
efficiency of the instruction. It is, therefore, necessary to 
inquire into the use of norms in indirect measurement, and 
into the nature of the various types of norms and their rela- 
tion to achievement and instruction. 


: The phrase “efficiency of a school” is used in this chapter in a re 
stricted sense. No account is taken of the educational investment. 
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The use of norms illustrated: two variables. One of the 
simplest illustrations of the use of norms is that of the stand- 
ard, or normal, temperature of the human body. Here, there 
are just two variable quantities. The temperature of the 
human body, which may be considered the dependent vari- 
able, and the general physical condition, the independent 
variable. The state of a person’s health is very closely re- 
lated to the temperature of his body. As the state of one’s 
health changes there is a corresponding change in one’s tem- 
perature. The temperature which accompanies perfect 
health is taken as the norm. Any deviations from this 
standard temperature indicate the presence of a pathologi- 
cal condition which needs medical attention. In the case 
of temperatures above the norm, we commonly express the 
deviations as so many degrees of fever. A person who has 
three degrees of fever has a temperature which is three de- 
grees above the norm. The deviation of one’s temperature 
from the norm is a measure of his general physical condition, 
but the measurement of the temperature of the human body 
without an appropriate norm with which to compare it 
would furnish no indication of the state of one’s health. 
The existence of a norm thus makes possible the measure- 
ment of one’s general physical condition, a thing which we 
are not able to accomplish by direct methods. 

Three variables. Norms are also used in situations that 
involve three variable quantities. One of the quantities is 
considered the dependent variable, and the other two are 
treated as independent variables. ‘The dependent variable 
depends upon the other two for its value. Any change in 
the magnitude of either of the independent variables pro- 
duces a corresponding change in the magnitude of the de- 
pendent variable. If one of the independent variables is 
easily measured by the direct application of an appropriate 
instrument, norms may be stated which define the amount 
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of the dependent variable which should, or on the average 
does, accompany certain amounts of this independent vari- 
able. The deviation of the dependent variable from these 
norms measures indirectly the other independent variable. 

This type of situation may be illustrated in the field of 
physical measurements by the weight, volume, and quality 
of wheat. The weight is the dependent variable. The vol- 
ume and quality are the independent variables. The weight 
and volume are easily measured in a direct way. This is 
not true of the quality. We may observe that the quality 
is poor and that the grains are shriveled and have a poor 
color. Or we may observe that another lot of wheat ap- 
pears to be excellent in quality. This, however, is not quan- 
titative measurement of the quality of the wheat. In the 
purchase and sale of this commodity the quality is an im- 
portant consideration. Hence, it is highly desirable to se- 
cure a quantitative measure of it. This is accomplished by 
means of the deviation of the weight of a sample having a 
given volume from the standard weight for that volume. 
The norm for wheat is that 2150.42 cubic inches (a bushel 
by volume) should weigh 60 pounds. A bushel of wheat 
does weigh this number of pounds if the quality approxi- 
mates the average. If the weight of a bushel (by volume) 
of wheat deviates from 60 pounds, the amounts of the de- 
viation is an index of the quality. For example, if a bushel 
of wheat weighs only 58 pounds its quality is two units be- 
low standard. If a bushel of wheat weighs 63 pounds its 
quality is three units above standard. Hence, by the use 
of a standard weight for a given volume we are able to se- 
cure an objective measure of the quality of wheat. 

Norms in more complex situations. In educational meas- 
urements the situations in which we need to use norms are 
even more complex, due to the fact that instead of only two 
independent variables there are three or more. The difficul- 
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ties that are introduced by having more than two independ- 
ent variables may be illustrated in the field of physical 
measurements of children. The weight may be taken as 
the dependent variable, and the height, age, and general 
physical development as the independent variables. If 
standard weights are determined for various heights, devia- 
tions from these norms are composite indices, or measures, 
of both age and general physical development. 

This, though, is unsatisfactory. The age of a child is 
not subject to control, and furthermore it is easily measured. 
What we desire is a measure of the general physical devel- 
opment, which is not easily obtained by a direct method. 
It, therefore, becomes necessary to do one of two things. 
One solution of the difficulty is to establish a separate set of 
standard weights based on height for each age. The other 
possibility is to show that age does not make a significant 
contribution to the weight of a child which is not already 
contributed by his height, and hence may be neglected as an 
independent variable. If this is done, standard weights es- 
tablished for each height would be satisfactory, and any 
deviation of the weight of a child from the standard weight 
for his height would be an index or measure of his general 
physical development. 

The type of norms required for evaluating the efficiency 
of a school. A pupil’s ability of which his performance, or 
more strictly speaking his score, is a measure depends upon 
his school grade (amount of instruction), general intelligence, 
and instruction. This last factor includes quality of instruc- 
tion, course of study, the pupil’s attitude toward his work, 
and other more subtle elements, all of which are subject to 
partial or complete control by the school. The effective- 
ness of the instruction is affected by the way in which the 
school is organized for instructional purposes. There are 
other factors, such as chronological age, general physical 
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condition, home environment, sex, race and nationality, 
which may be listed as contributing to the achievement of 
the pupil. The effect of these factors upon achievement is 
not known with much certainty, but it appears that their 
contributions are relatively slight except in extreme cases. 
For these reasons we shall omit them in the following dis- 
cussion. ‘To include them would greatly complicate our 
analysis without adding anything to the practical value of 
the conclusions. 

Achievement is the dependent variable. School grade, 
general intelligence, and efficiency of instruction are the in- 
dependent variables. If norms of achievement are stated 
with reference to school grade, the deviation of a score from 
the grade norm is a measure of the combined influences of 
general intelligence and instruction. Thus, the interpreta- 
tion of the deviation of a score from the norm is ambiguous 
because in the case of a pupil who is below standard we can- 
not know whether this condition is due to his mental age, 
over which the school has no control, to the quality of the in- 
struction, over which the school has control, or to a combina- 
tion of the two factors. A similar statement can be made 
with reference to the deviations of class scores from grade 
norms. 

Relative influence of general intelligence and school 
grade upon achievement. ‘There is an increase in the aver- 
age mental age of pupils from grade to grade. In order to 
study the relative influence of general intelligence and school 
grade upon the achievement of pupils,! the Illinois General 
Intelligence Scale and the revised form of Monroe’s Stand- 
ardized Silent Reading Tests, were given to about seven thou- 
sand school children in sixteen Illinois cities. These two 
tests were given to all of the pupils in grades III to VIII. 


1 Monroe, Walter S. The Illinois Examination. Bulletin No. 6, Bureau 
of Educational Research, University of Illinois Bulletin, vol. xrx, No. 9. 
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There is in general an increase in the average rate and com- 
prehension from grade to grade for the same mental age, 
and also from a lower to a higher mental age within the same 
school grade. The average increase in achievement from 
grade to grade for pupils of the same mental age was found 
to be .3 for comprehension, and 5.7 for rate of silent reading. 
The average increase in achievement corresponding to an 
increase of one year of mental age for pupils belonging to 
the same school grade was found to be 1.1 for comprehen- 
sion and 9.4 forrate. Thus, the average increase for one 
year of mental growth is over three times as great for com- 
prehension, and nearly two times as great for rate as the 
corresponding increases from one school grade to the next. 
Hence, we are justified in saying that on the basis of these 
facts the mental age of a pupil makes a much greater con- 
tribution to his achievement in silent reading than does the 
school grade in which he is placed. This being the case, 
our interpretation of achievement scores in silent reading 
will be very misleading unless the mental age of a pupil or 
a group of pupils is taken into account. 

It must be borne in mind that the above investigation in- 
cluded only silent reading. It is possible that achievement 
in other school subjects is not related in this way to general 
intelligence. It is also possible that the repetition of the 
investigation in the field of silent reading with other tests 
would not yield similar results. It is, however, very un- 
likely that the results would be sufficiently different to indi- 
cate the opposite conclusion. 

The diagnosis of groups of pupils. Deviations of the 
class score from the grade norm measure the combined effect 
of general intelligence and instruction. Since mental age 
has been shown to be a potent factor in determining achieve- 
ment, the deviations fail to give definite information con- 
cerning the effectiveness of the instruction which the class 
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has received. If the average mental age of the class is con- 
spicuously above the average for grade, their class score may 
be up to the grade norm but the instruction may be lacking 
in effectiveness because, when the intellectual equipment is 
considered, this class should have an average achievement 
score notably above the grade norm. The corresponding 
statement can be made for a class whose average mental 
age is below that for their school grade. Hence, grade 
norms are not satisfactory for evaluating the efficiency of the 
instruction given to a class. The use of mental age norms 
for this purpose will be considered after our discussion of 
the diagnosis of individual pupils. 

The diagnosis of individual pupils. There are large 
individual differences in achievement within a school grade. 
This is true even if the pupils belong to a single class. Thus, 
the deviations of some individual scores from the grade norm 
will be large. In a number of cases they will be greater than 
the difference between the norms for two successive grades. 
These deviations are measures of the composite effect of gen- 
eral intelligence and instruction. Since they are large, the 
influence of either general intelligence or instruction or both 
must be large. 

Within a given class the pupils have received approxi- 
mately the same instruction, but it may not have been 
equally effective in engendering achievement. Pupils differ 
in their responses to particular kinds of instruction, as well 
as in other respects. Thus, it appears reasonable that the 
effectiveness of the instruction makes a substantial contri- 
bution to the deviations from the grade norm. 

On the other hand, the application of a general intelli- 
gence test reveals very large differences in the mental ages 
of the pupils belonging to a given class. Differences of four 
or five years are frequently found. Since mental age is de- 
fined as capacity to learn, it is obvious that differences in 
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the mental ages of pupils are influential causes of the devia- 
tions of the achievement scores from the grade norms. 
Thus, the deviation of a pupil’s score from the grade norm is 
a very poor measure of the effectiveness of instruction. It 
is not too much to state that in the diagnosis of individual 
pupils grade norms are of little value. Little information 
concerning the effectiveness of the instruction is given by 
the deviation of the pupil’s score from the grade norm, un- 
less the mental age of the pupil is also considered. 

Grade norms for different mental ages. One way of tak- 
ing account of the mental age of pupils and classes in the 
use of grade norms would be to establish separate grade 
norms for the different mental ages that are found within 
each grade. This plan would give a very complex system of 
norms. For the interpretation of individual scores there 
would need to be at least five norms for each grade. The 
disadvantage of this plan lies in its complexity. 

Mental-age norms superior to grade norms. The fact that 
mental age appears to make a larger contribution to a pupil’s 
achievement than his grade placement suggests the establish- 
‘ment of mental-age norms to take the place of grade norms. 
Such norms can be derived by grouping the scores of pupils 
on the basis of mental age, rather than on the basis of school 
grade. This will involve a greater amount of labor in the 
derivation of norms, since it will be necessary to give to each 
pupil a general intelligence test as well as the achievement 
test. However, the procedure involves no serious difficulty. 

The deviation of an achievement score from a mental-age 
norm will be an index of the composite influence of school 
grade and instruction. The investigation referred to above 
indicated that the school grade makes less contribution to a 
pupil’s achievement than his mental age. Hence, mental 
age-norms are more satisfactory than grade norms in evalu- 
ating the efficiency of instruction. 
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Comparison with mental-age norms by ratio or achieve- 
ment quotient. Instead of comparing an achievement score 
with the norm by subtraction, to obtain a deviation, com- 
parison may be made by forming a ratio. ‘The achieve- 
ment score divided by the norm will give a quotient which is 
essentially the per cent of the norm which has been achieved. 
This is called the achievement quotient. + 

The achievement quotient furnishes a simple method of 
allowing for the effect of the grade placement of the pu- 
pil upon his achievement score. An achievement quotient 
(A.Q.) of 100 indicates that the pupil’s achievement is just 
up to his mental-age norm. If the grade placement of the 
pupil makes an appreciable contribution to his achieve- 
ment, we would expect that a pupil of a given mental age in 
the sixth grade should have a higher achievement quotient 
than a pupil of the same mental age in the fourth grade, pro- 
viding the instruction and other factors, except school grade, 
conditioning achievement were equal in the two cases. If 
this is found to be true, standard achievement quotients 
may be set up for each school grade. Such a dual system of 
norms will be more convenient than a norm for each mental 
age within each school grade. It has, however, been shown 
for silent reading and the operations of arithmetic that the 
standard achievement quotients are approximately constant 
in the different grades.? 

Advantage of mental-age norms. Certain advantages 
may be enumerated for mental-age norms. First, mental- 
age norms emphasize the general intelligence of pupils. This 
is a good thing, since general intelligence makes a large con- 
tribution to the pupil’s achievement. Second, the achieve- 
ment quotient which is obtained by comparing a pupil’s 
achievement score with his mental-age standard is a much 
better index of the effectiveness of the instruction which the 

1 See Chapter VII. | 2 See page 159. 
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pupil has received than the deviation of his score from the 
grade norm. The achievement of each pupil is compared 
with his own norm rather than with the norm for the group to 
which he happens to belong. Third, since mental-age norms 


are 


essentially individual, rather than group, emphasis is 


based upon each pupil doing his best. The achievement of 
the other members of his group has no effect upon his score, 
or upon the norm with which his score is compared. 


onl 


QUESTIONS AND TOPICS FOR INVESTIGATION 


. How are the norms for our educational tests derived? 
. How should we interpret norms with reference to educational objec- 


tives? 


. Summarize the merits and limitations of the different types of norms. 
. Why is it necessary to control conditions in the administration of 


educational tests? 


. Explain how certain magnitudes may be measured indirectly through 


the use of norms. 


. When a test has been standardized, are we justified in considering 


these norms permanent or will it be necessary to restandardize a test 
frequently? 


. Should we expect to have norms for each month of the school year? 


Why? 


CHAPTER IX 


HOW TO MAKE A CRITICAL STUDY OF AN 
EDUCATIONAL TEST 


The general outline for a critical study of an educational test. 
The need for a critical study of an educational test arises 
both in the construction of a new instrument and in the sci- 
entific selection of one of the available tests for use. A re- 
fined use of a measuring instrument requires that its limita- 
tions — i.e., its deviations from the necessary requirements 
— be known. The principal items in a critical study of an 
educational test refer to the extent to which the requirements 
of test construction, stated in Chapter IV, are satisfied. 
The outline for the critical study of a test which follows 
has been prepared to show the questions which should be an- 
swered. Some of the questions call for facts; others require 
inferences from information which, in some cases, is in- 
complete. The most important question is that of validity, 
or the constancy of the functional relation between the scores 
and the abilities specified by the function. Several of the 
items called for in the second and third divisions of the out- 
line derive their significance from their use in the consider- 
ation of validity. For many of our present tests it is not 
possible to answer several of the questions in the outline be- 
cause the necessary information is not available. When 
applying the outline to such tests the study must, neces- 
sarily, be limited to the questions for which information can 
be secured, unless it is possible to supplement it by original 
research. Following the outline the questions will be ex- 
plained, and the method of answering them will be described 
and illustrated. The pages in the text on which the explan- 
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ations of the main headings of the outline begin are indicated 
in parentheses. 


OUTLINE FOR MAKING A CRITICAL STUDY OF A TEST 


I. Facts of title (p. 185). 
1. Author. 
2. Exact title. 
3. Date of first publication or use. 
4. Duplicate forms and parts. 
5. Used in what grades? 
I. Nature of pupil’s performance (p. 185). 
1. What the pupil does. 
2. How the exercises of the test were constructed and 
selected. 
3. Conditions under which performance is given. 
a. Explanation of test to pupils. 
b. Directions as to methods of work. 
_¢. Time allowance. 
d. Distractions. 
4. In the case of a scaled test, are the exercises equally 
spaced on the scale of difficulty? : 
IM. Description of pupil’s performance (p. 186). 
1. How is the score computed? (This is to include rules 
for scoring, and derived scores.) 
a. Individual pupil. 
b. Class or larger group. 
2. What dimensions are described separately? 
8. What dimensions are described in combination, and 
what is the nature of the combination? 
‘4, What dimensions are not described in the pupil’s 
score? Are these constant for all pupils? 
5. Does zero mean “‘not any” of the ability measured? 
; Is the unit constant? 
IV. Function of the test, or specification of the abilities the test 
is designed to measure (p. 186). 
V. Validity of the test, or the constancy of the functional rela- 
tion between the scores and the abilities specified by the 
function (p. 188). 
1. Objectivity in describing performances (p. 196). 
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2. Reliability (p. 201). 

a. Coefficient of reliability (712) (p. 202). 

b. Index of reliability (73) (p. 206). 

c. Probable error of measurement (P.E.y) (p. 207). 

d. Coefficient of correspondence (p. 218). 

e. Overlapping of successive grade groups (p. 219). 

3. Discrimination (p. 219). 

a. Does the distribution of measures agree with the 
normal curve? 

b. Are differences shown between groups which are 
known to differ in ability? 

c. Into how many groups is a typical class divided? 
Is this sufficient to discriminate properly between 
the members of a class? 

4. Comparison with criterion measures (p. 221). 

a. Teachers’ marks. 

b. Measures yielded by other tests. 

ce. Composite test scores. 

5. Inferences concerning validity, based upon the struc- 
ture of the test and its administration (p. 226). 

a. Do the content of the exercises and the structure 
of the test-appear to be consistent with its func- 
tion? 

b. Do all pupils have an opportunity to demonstrate 
their abilities? 

c. To what extent are testing conditions controlled? 

d. Is the variation of abilities, other than those 
being measured, reduced to a minimum? 

VI. Validity of significance (p. 227). 

1. Do the traits measured have educational significance? 

2. Are the measurements expressed in terms of signifi- 
cant dimensions? 

3. Do the norms form appropriate educational objec- 
tives? 

4. To what school activities is the function of the test 
applicable? 

VII. Norms (p. 229). 

1. Types of norms available. 

a. Grade. 

b. Chronological age. 

c. Mental age. - 
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2. Representative character of scores on which norms are 
based. 
a. Number of scores. 
b. Type of population groups. 
ce. Time of school year for which norms are stated. 
3. Effect of acquaintance with form of test upon stand- 
ards. 
4, Equivalence of duplicate forms. 
VIII. Practical considerations (p. 230). 
1. Time required for giving and scoring. 
2. Cost of test materials. 


We shall now consider each of these topical headings, in 
the order given above. 


T-IV. Tirte, Nature, DEscrIPTION, AND FUNCTION 


Facts of title. The questions under the first general head 
require no explanation, since they call for simple facts. 

Nature of pupil’s performance. In every test the pupil 
does something: adds columns of figures and writes the sums, 
writes answers to questions from memory, makes a drawing, 
writes sentences from dictation, answers questions based 
upon a text to which he can refer by underscoring a word, 
completes sentences by writing missing words, crosses out 
superfluous words or other symbols, etc. The first question 
asks for a statement of what the pupil does. It is frequently 
desirable to supplement the description by quoting a sam- 
ple of the exercises. 

The second question inquires into the procedure followed 
by the test-maker in determining the particular exercises 
the pupil is asked to do. For example, in spelling, the ques- 
tions are, ‘‘How were the test words selected with reference 
to both the course of study and the difficulty of the words?” 
** What were the considerations which determined the man- 
ner of presenting the words?” In silent reading, the ques- 
tions are similar, ‘“‘ What were the considerations which 
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lead to the selection of the material to be read?” “Is this 
material of a particular type? If so, why?” “What con- 
siderations lead to the choosing of the type of exercises 
used?” ‘How were they constructed?” “Were all the 
exercises that were constructed used? If not, how were the 
ones used selected?” The particular questions to be an- 
swered vary somewhat with the type of test and with the 
subject-matter field, but these two cases illustrate their 
general character. 

The meaning of the third question is indicated by the 
sub-questions. The first two of these distinguish between 
explaining to the pupil what he is to do—add a column of 
figures, cross out words, write from dictation, etc. — and 
how he is to do it — rapidly, carefully, check each exercise, 
etc. Directions to pupils concerning their methods of work 
may materially affect their performances. The directions 
to examiners may also provide for controlling distractions. 
This is particularly important in individual testing. 

Description of pupil’s performance. The description of a 
pupil’s performance has been discussed in Chapters V, VI, 
and VII. After a study of these chapters the questions un-: 
der this division of the outline should not need explanation. 
The first one calls for facts which are to be derived directly 
from the test and the directions which accompany it. Qcca- 
sionally, users of tests have departed from the directions for 
scoring specified by the author. Any variations of the 
method for computing the score should be noted. 

Function of the test. The statement of the function of : a 
test specifies the abilities that it is designed to measure. 
These are the abilities with reference to which the validity 
of the test is to be judged. Any function may be announced 
for a test, but apparent inconsistencies between the struc- 


1 Thorndike, E. L., and Courtis, S. A. “Correction Formule for Ad~= 
dition Tests”’; in Teachers College Record, vol. xx1, pp. 1-24 (January, 1920), 
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ture and content of the test and the announced function 
must be considered in judging its validity. In such a case 
experimental evidence must be produced to show that the 
announced function is fulfilled in spite of these inconsisten- 
cies. Any test measures something, although it may do so 
very inaccurately. A test may be found not to have the 
function announced for it, but to fulfill another function with 
a high degree of accuracy. Therefore, one of the results of 
a critical study of a test may be a redefinition of its function. 
For many of our existing educational tests the function 
has been stated only in very general terms. In some cases, 
no statement is available except that implied in the title of 
the test. In making a critical study of such tests, a more 
exact statement of the function should be formulated by 
adding the implications of the structure and content of the 
test. For example, the title, Courtis Standard Research 
Tests in Arithmetic, Series B, suggests that the function of 
this series of tests is to measure arithmetical ability. An 
examination of the structure and content, together with a 
knowledge of the nature of ability in the field of arithmetic, 
indicates that its function is restricted to the measurement 
of the arithmetical abilities required for performing the four 
fundamental operations with integers. Abilities in the fields 
of common fractions, decimal fractions, and problem solv- 
ing are not included. The title, Monroe’s Standardized 
Silent Reading Test, suggests that the function of this test 
is to measure the ability to read silently. An examination 
of the test shows that it consists of exercises which require 
that the pupil answer questions based upon the paragraphs 
read. The material to be read is not connected, but it 
consists of separate exercises. These conditions suggest that 
a more appropriate statement of the function would be “to 
measure the ability to read silently simple unconnected 
paragraphs for the purpose of answering questions.” 
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The statement of the function of an educational test 
should include specifications with reference to: 

1. The particular abilities measured. 

2. The type of growth measured. (Two types of growth 
have been recognized. See page 66.) 

3. The type of measures the test is designed to yield — 
general or specific. 


V. Vaurity or THE TEST 


Under the head of validity we inquire into the degree of 
constancy of the functional relation existing between the 
scores yielded by the test and the abilities specified as being 
measured in the statement of its function. A constant rela- 
tion means that, for any change in score from pupil to pupil, 
there is a corresponding change in the abilities specified. 
Since we use common norms for interpreting all scores, we 
must investigate this constancy of relation with reference to 
pupils tested by different examiners at different places and 
different times. 

The validity of physical measurements. In studying the 
validity of the measurements of physical objects, as, for ex- 
ample, those yielded by a yardstick, we have to consider 
only two questions, (1) the accuracy of the length of the 
yardstick and its subdivisions into inches and fractions of 
inches, and (2) the accuracy with which it is used by com- 
petent persons. The first question can be answered by com- 
paring the yardstick with a recognized standard instrument. 
The accuracy of the measures yielded by the instrument 
may be studied by having a number of competent persons 
make measurements from the same object. The fluctua- 


1 If the measurements are made with a high degree of precision, it is 
necessary to consider, also, the constancy of the measuring instrument. 
For example, temperature, and, in the case of certain materials, humidity, 
affect the length of a yardstick. 
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tions of these measurements from the average constitute an 
index of their accuracy. These fluctuations may be used as 
a basis for predicting the magnitude of the departures from 
a constant functional relation between measures and the 
magnitude of the objects which may be expected in the 
measures secured by competent persons.! 

Determination of validity of an educational test complex. 
The determination of the departures from a constant func- 
tional relation between scores yielded by an educational test 
and the abilities specified by its function is more complex. 
Ability is measured through a performance, and the per- 
formance is described by a score. A brief consideration of 
the nature of the functional relation between the score and 
ability will assist in understanding the nature of the prob- 
lem with which we are dealing in a study of the validity of an 
educational test. The functional relation of performance to 
ability has been discussed in Chapter IV. A pupil’s perform- 
ance was shown to depend upon a number of factors other 
than the ability being measured. Two illustrations will 
indicate the magnitude of the variations from a constant 
functional relation which occur under testing conditions 
which appear to be constant to a high degree. It should be 
noted that both of these illustrations deal only with the de- 
parture from a constant functional relation, in the case of a 
single pupil, when tested at different times. Only by im- 
plication do they indicate the variations from pupil to pupil. 

Variability of performance in spelling. Ashbaugh ? re- 
ports a study in which the same spelling test was given to 
the same pupils three times within fifteen minutes, and un- 
der conditions which were highly constant. The resulting 


1 The question concerning what the yardstick measures is not raised. 
It is assumed that it measures the straight-line distance between two points, 
and this assumption agrees with our experience. 

2 Ashbaugh, E. J. “Variability in Spelling”; in School and Society, vol. 
Ix, pp. 93-98 (January 18, 1919). 
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performances were not consistent. Certain pupils would 
spell a word correctly the first time and miss it the other two. 
In fact, there were all possible combinations of correct and 
incorrect spellings of the same word. The following table 
shows the per cent of pupils who gave variable perform- 
ances when given three opportunities, during a fifteen- 
minute interval, to spell each of twenty words. 
Per cent of pupils who gave variable performances: 


Gradél re. thc cre saaects VEeVil Vit 
Towa Cityes i ecnacse aes cs 25 2 18 
Hibbing: sraice Rikon 22 25 26 


Approximately one fourth of the pupils failed to maintain a 
constant functional relation between performance and spell- 
ing ability during this fifteen-minute interval. 

Variability of performance in arithmetic. Tests 7 and 9 of 
Monroe’s Diagnostic Tests in Arithmetic were given to 
about ninety fourth-grade pupils; tests G and K of the 
Cleveland Survey Arithmetic Tests were given to about 
eighty sixth-grade pupils; and two tests in the field of com- 
mon fractions, one in division and the other in addition, 
were given to about ninety eighth-grade pupils. Test 7 con- 
sists of single column addition examples, thirteen figures to 
the column. Test 9 involves subtraction, the subtrahend 
in each instance being a three-place number. Test G is mul- 
tiplication, the multiplier in each case being a single digit. 
Test K is long division, with a two-figure divisor. In no 
case does the second figure of the divisor exceed 3. In the 
addition of fractions it is necessary to find a common denom- 
inator which is the product of the denominators of the two 
fractions. In the division test the fractions are simple, but 
some reduction is necessary to bring the results to their low- 
est terms. In the first four tests the exercises are considered 
to form equal work units, although they are not identical. 
They are identical in gross structure, and the usual use of 
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such tests implies the assumption that all of the exercises call 
for the same abilities. In the last two tests an attempt was 
made to have the exercises as nearly equivalent in difficulty 
as possible. In giving these tests, the usual directions were 
not followed. All pupils, except the very slowest, were given 
an opportunity to complete their work. Thus, no measure 
of the rate of work was obtained. Only accuracy was con- 
sidered. 

Variability in accuracy shown by a table. The perform- 
ance of each pupil was divided into sections of 3, 4, 5, and 
6 examples, each, for the purpose of studying the constancy 
of the accuracy. The amount of change in accuracy, from 
section to section, indicates one departure from a constant 
functional relation. These changes in accuracy ranged 
from an increase of one hundred per cent to a decrease of one 
hundred per cent. A number of pupils maintained the same 
accuracy through two sections of their performance. In 
Table VI, the per cent of pupils who did this is given for the 
first four sections of each performance. The table is to be 
read, as follows: On the addition test in grade four, taking 
three examples in the group, 48 per cent of the pupils main- 
tained the same degree of accuracy in the second group of 
examples that they exhibited in the first group; 41 per cent 
maintained the same degree of accuracy in the third group 
as they exhibited in the first group, and so on. It will be 
noted that the per cent of pupils who maintained the same 
degree of accuracy varies with the different tests. Since 
these tests were given in different grades, some of this change 
may be due to differences in the maturity of pupils. How- 
ever, it is likely that it is due, in part, to the operations per- 
formed and, in part, to differences in the length of the exam- 
ples. It will be noted, also, that there is a small decrease in 
the per cent of pupils who maintained the same degree as the 
number of examples in the group is increased. This suggests 
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that there is some relation between the consistency of per- 
formance and the length of the performance. 

Both illustrations point to the conclusion that the func- 
tional relation between performance and ability is not con- 
stant for many pupils, even when the testing conditions are 


TABLE VI. PER CENT OF PUPILS WHO MAINTAINED THE 
SAME ACCURACY OF PERFORMANCE THROUGH TWO 
GROUPS OF EXAMPLES 


(Number of examples | ¢, aa |tJ-1|I1I-I|T1I-1|IV-I|IV-I1|\IV-IIl\ Av. 


in groups 

Addition $| IV|48] 41| 89 | 38] 88 | 40 40.7 
4 36 | 32] 31 33.0 
5 31| 32] 937 83.8 
6 30 30.0 
Subtraction 8| IV] 49] 54] 56 | 51] 52 52 | 52.3 
4 24| 41} 51 | 43] 50 42 | 41.9 
5 46 | 44] 43 | 46] 62 39 | 46.7 
6 41{ 41] 88 | 45] 33 48 | 41.0 
Multiplication 3] YI} 48] 46] 56 | 41] 52 43 | 47.5 
4 41 | 88] 87 | S@| 26 45 | 36.5 
5 31 | 37 | 35 39 | 29 35 | 34.3 
6 27 | 25] 33 28.3 
Division © S| VIje6s| 65] 77 | 72] 67 75 | 69.8 
4 65 | 67 | 72 69 | 66 77 | 69.3 
5 62 | 63 | 67 64] 77 67 | 66.7 
6 60 | 66] 69 | 62} 58 58 | 62.1 

Addition of 
fractions 3 | VIII |73| 71] 77 | 64] 78 70 | 72.1 
4 69 | 68| 67 | 61 | 70 67 | 66.1 
5 68 | 55] 58 | 44] 52 60 | 56.1 
6 57 | 43 | 51 50.3 

Division of 


fractions VIII | 65 | 57 | 69 | 55] 64 61 | 61.8 
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as near constant as can be made. In the general use of 
educational tests, the testing conditions are more variable. 
Different examiners administer the tests, and the tests are 
given at different times and in different places. Further- 
more, when we consider constancy of functional relation 
from pupil to pupil, instead of comparing each pupil with his 
own record, we would expect to find still greater variations. 
It, therefore, appears that the functional relation between 
performance and ability may be eharacterized as variable. 
We have assumed in this discussion that a pupil’s ability in 
a given field was constant throughout a short time interval. 
It may be contended that it is variable, but in this case we 
are concerned with his average ability and the above state- 
ments apply. 

The functional relation of performance to ability. In 
Chapter IV, page 68, the relation of performance to the fac- 
tors upon which it depends was represented by the equation: 


Prat Tie tes ues 2's Og As os Das» +> Le) 


in which P stands for the objective performance; aj, dg, As, 
. . . a, the abilities 1 being measured; and 2), 2g, #3, . . . 2, the 
other factors which affect the performance. The 2’s include, 
as general factors, such things as the effort which the pupil 
makes, his physiological condition, emotional status, and 
degree of concentration. Other z factors, such as the writ- 
ing of the sums in a test on addition combinations, or spell- 
ing, vocabulary, and other language abilities in a silent read- 
ing test requiring reproduction, are sometimes involved.? 


11In this connection it is, perhaps, helpful to distinguish between the 
ability a pupil possesses but may not exercise completely because he makes 
little effort, and the ability which functions in the production of the perform- 
ance. The former we may call his potential ability. The ability which is 
active is his kinetic ability. This is what we measure. 

2 It is sometimes possible to define ability so that the element of writing 
is included in it. In the case of the operations of arithmetic, it seems desir- 
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The functional relation between score and performance. 
The score obtained from a given performance depends upon 
certain factors other than the performance. The unit used, 
its constancy, the location of the zero point, the method of 
scoring, the training of the scorer, and the objectivity of the 
scoring are potent factors in determining the score. If the 
method of scoring is highly objective, the training of the 
scorer will affect the scores only slightly; but, in the case of 
such school subjects as handwriting or English composition, 
the training of the scorer is a very potent factor. Employ- 
ing again mathematical symbolism, the relation between the 
score and performance may be expressed by the equation: 


S = F(P, yr, Yas Ys - + + Yn) 


in which 4}, Ya, Ys, «+ » Yn include such factors as those just 
mentioned and human fallibility. 

The functional relation between the score and ability. 
The functional relation existing between the score and the 
ability can be expressed by the following equation, in which 
the symbols have the meaning already given to them: 
S=F [f (a, Ag, Ag, + + «Any Vip Ta, Ug, -- - Lp) Yis Yas Yar +++ Ynl 
This relationship may be expressed more briefly by the 
equation: 


S = ¢(a, 2, y) 


In this equation a represents the total of the abilities, and x 
and y the totals of the two classes of factors. 

Methods of determining validity. The most direct method 
of determining the validity of a given test would be to secure, 
independently of it, measures of the abilities it is designed to 
measure which are known to be highly accurate, and then to 
able to do this except, perhaps, for the fundamental combinations and other 


simple types of exercises which are usually performed mentally in the doing 
of a more complex type of example. 
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compare the measures yielded by the test in question with 
these. The lack of agreement would constitute an index of 
the validity of the test. However, this method is not possi- 
ble, in general. The difficulty is that we are not able to se- 
cure independent measures of the abilities which are recog- 
nized as highly accurate. In fact, standardized objective 
tests are proposed as instruments which will yield more accu- 
rate and, hence, more valid measures of mental abilities than 
we are able to obtain by other means. We are, therefore, 
compelled to study the validity of the function of most tests 
by methods which are indirect and open to certain limita- 
tions. 

In a few instances there are available independent meas- 
ures which are recognized as having a high degree of validity. 
For example, in the case of intelligence tests, the Stanford 
Revision of the Binet Test is considered to yield measures 
of intelligence which have a high degree of accuracy. An 
index of the validity of another intelligence test may be 
obtained by comparing the scores yielded by it with the cor- 
responding Binet scores. A common ‘method of comparison 
is by correlation; and the coefficient of correlation between 
the two sets of scores becomes an index of the validity of the 
test. Some tests, principally tests of general intelligence, 
have for their function the measurement of a pupil’s future 
success in a given field. Such tests are called prognostic. 
The measures of success in school work are definite, even if 
they are not always true measures of the pupil’s real accom- 
plishmenis. Since they are definite, they may be used as 
criteria for determining the validity of the function of such 
prognostic tests.! 


1See Kelley, T. L. ‘The Reliability of Test Scores”; in Journal of 
Educational Research, vol. 1, pp. 370-79 (May, 1921), for a discussion of 
the need for knowing, also, the reliability of a test because the available 
criteria are not perfectly reliable. 
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1. Objectivity in describing performances 


The “personal equation” in testing. One source of de- 
parture from a constant functional relation between the 
score and the ability is the users of the measuring instru- 
ment. Different examiners may not secure the same per- 
formances from the same pupils due to differences in the 
administration of the test. Since numerical measures of 
these differences cannot easily be obtained, the conditions 
under which variations in the administration of the tests 
are to be expected will be considered under the head of “‘in- 
ferences concerning validity.”” The description of the per- 
formances of pupils offers another opportunity for differences 
in the scores obtained by different examiners. In such a 
field as the operations of arithmetic the scoring can be made 
objective, except for chance errors on the part of scorers. 
Only one answer can be correct. If credit is to be given only 
for answers entirely correct, there can be no differences of 
opinion concerning when to allow credit and when to give 
no credit. In some other subject-matter fields, exercises 
have been constructed which permit of only one correct an- 
swer. When this is done, the test is objective with reference 
to the scoring. 

When the scorer is asked to exercise judgment concerning 
the quality of a performance, different scorers will give differ- 
ent scores. This occurs in the use of a quality scale in rating 
samples of handwriting and in describing reproductions of 
material read. It also occurs in scoring answers to certain 
types of questions. The subjectivity of the description of 
such performances may be illustrated by the measurement 
of English compositions. W. W. Theisen ! reports the rat- 


1 Theisen, W. W. “Improving Teachers’ Estimates of Composition 
Specimens with the Aid of the Trabue Nassau County Scale”; in School 
and Society, vol. vit, pp. 148-50 (February 2, 1918). 


CRITICAL STUDY OF A TEST 197 


ing of a number of compositions by each of fifteen teachers. 
The Nassau County Supplement to the Hillegas Scale was 
used as an instrument to assist in describing the quality of 
these pupil performances. The values assigned to the com- 
positions of this scale range from 0.0, 1.1, 1.9, to 9.0. 


TABLE VII. TEACHERS’ RATINGS OF COMPOSITIONS USING 
NASSAU COUNTY SUPPLEMENT TO THE HILLEGAS SCALE 
(Arranged from Theisen’s Report) 


Compositions 

Teacher 
F-3| C-7| B-8| B-4| I-7 | C-#| B-6| C-3| B-10| G-3|F-9| A-9 
Desers 8.8/8.0/}5.0/1.9]7.2|}8.0|}9.0/6.0|] 5.0 |2.815.0] 6.0 
LGA 0.0|6.0|3.8/2.8)8.0|7.2|7.216.0] 3.8 |}2.815.0] 3.8 
25 pete AA 2.8|8.0)7.2)8.0|8.0/8.0/9.0/7.2] 5.0 |3.8|3.8]| 5.0 
Ae esi 1.9}8.0)7.2|2.8)8.6|8.0)9.0|7.2| 5.0 |3.2|5.2| 5.0 
SIS, TRG 2.817.2|3.8/2.819.0|8.0|/9.0}7.2) 6.0 |3.8|6.0| 5.0 
OMe orice 1.9| 6.0/3.8) 2.8 |'6.0}8.0/8.0;6.0) 5.0 | 2.8/2.8] 3.8 
Tie ea 1.9|7.2)6.0/2.8)8.0)9.0|7.2|3.8; 5.0 |2.8|5.0| 1.9 
Bei 0.0/}9.0|6.0/2.0|8.2|8.0/9.0)7.3| 5.4 | 3.8/6.0] 6.0 
C! Neeser 1.9|8.0]6.0|0.7/8.0/9.0/9.0/5.8) 5.0 |1.2|)4.2] 1.9 
AOE eee 2.8|7.2}6.0/}2.8/8.0/9.0}9.0/7.2] 6.0 |3.8]2.8] 2.8 
yh Pe $.8|)7.2/2.8/2.817.819.0|7.2/5.0] 5.6 |3.8/2.8) 6.0 
BC ype 1.9|7.217.2|2.8|7.218.0/6.0)8.0| 3.8 |6.0/6.0] 2.8 
1 ae ae 2.1/9.0)}6.0|)0.7|7.2)8.017.2)9.0|] 6.0 |}2.815.0] 5.6 
i ete ee 1.2|7.2)6.0|3.819.019.0/}8.0/9.0| 4.4 |3.8] 7.2] 2.9 
eet $.8/8.0)7.2|2.8|8.0/19.0/8.0)2.8| 8.8 |3.8|3.8) 5.0 

Standard 
value..| 2.0 | 7.4)3.6|]1.3/9.3|8.0/8.3|5.9} 4.7 |2.514.0| 3.2 


An examination of Table VII! clearly shows that these 
teachers did not agree on the description of the quality of any 
of the composition included in this study. The descriptions 
of composition F-3 range from 0 to 3.8. For composition 
C-3 the range is from 2.8 to 9.0. When it is remembered 


1 Values between the steps of the scale were assigned in some cases unless 
there are some errors in Theisen’s report. 
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that the total range of quality on this scale (rd 0. 0 to 9.0) 
is approximately that exemplified by the writings of elemen- 
tary school pupils, the differences in the ratings by these 
teachers become very significant. These ratings were made 
after the teachers had received only slight training in the use 
of the scale. A few hours of training will materially increase 
the objectivity of the descriptions of the quality of compo- 
sitions. 

Constant and variable errors. Subjective scoring in- 
volves two types of errors, constant errors and variable er- 
rors. A constant error results in scores which, in general, 
are too high or too low. A liberal attitude toward the per- 
formances of pupils will result in high scores. On the other 
hand, a conservative procedure will result in low scores. An 
indication of the presence of a constant error may be secured 
by comparing the averages of the two sets of scores assigned 
independently by two scorers to the same set of test papers. 
Any differences in their general policy will be reflected by a 
difference between these averages. However, the difference 
cannot be considered to be an exact index of the magnitude 
of the constant error because both persons may be inclined 
to be liberal in their scoring or both may be conservative, or 
one may be conservative and the other liberal. 

Variable errors are indicated by the fact that, in scoring 
one performance, Scorer A will assign a score of 90 and 
Scorer B a score of 75; but, in scoring a second performance, 
Scorer A may assign a score of 70 and Scorer B a score of 80. 
This may happen, although Scorer B is, in general, more 
liberal than Scorer A. In studying the variable errors it is 
necessary to isolate them from the constant errors. The 
latter do not affect the coefficient or correlation; hence, it 
may be used as an index of the magnitude of the variable 
errors. 


Constant and variable errors involved in scoring repro- 
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ductions. Table VIII gives data relative to both the con- 
stant and variable errors involved in the word-counting 
method of scoring reproductions. The scorers are repre- 
sented by letters. The numbers in the column headed 
“Difference of average scores”’ were obtained by subtract- 
ing the average of the scores assigned by the second scorer 
from the average of the scores assigned by the first scorer. A 


TABLE VIII. SUBJECTIVITY OF SCORING REPRODUCTIONS 
BY THE WORD-COUNTING METHOD 


Differ- 
Number emer 
Test Form | Grade of |Scorers aherage ts P.E. Est 

re scores 
Memory......... I IV 92 | Y-C| — 9.9] .89 | 4.5 
I IV Q7 Y-K | — 5.1 .92 3.4 
ve II IV 116 Y-C | — 2.0 .90 3.3 
sf VII 123 Y-K | — 7.5 Wi 5.5 
ed II VII 100 | YC} — 8.2] .84 | 3.9 
se II Vil 31 | Y-K/| + 4.1] .90 | 2.6 
Reproduction I IV 94 | L-K|+ 6.8] .97 | 8.1 
II IV 31 | L-C | — 1.6] .98 | 2.4 
fs II IV 68 L-K | + 4.7 .92 4.2 
ss I VII 117 M-F|} — .5 .96 9.2 
“ Il VII 1138 | FC |] — 6.0] .97 | 5.5 
Brown. ..e.ees:- Tidy iv | Aid| T-Mel 12-6 |...89.) 6.8 
“ II IV 110 | T-My| + 6.9 .88 4.8 
Sta h (No. 7 I VII 119 M-C | — 5.8 .97 2.6 
- NS Bens m | vit | 121 |M-C| — 2.0] .97 | 2.1 


positive difference means that the first scorer gave, on the 
average, higher scores than the second. A negative differ- 
ence has the opposite meaning. In some cases the differ- 
ence closely approximates zero, but in others it is relatively 
large. This indicates that for some scorers the constant 
error is relatively large. One is justified in asserting that, 
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on the basis of the possible constant error in the scores as- 
signed to reproductions by a single scorer, no reliable infer- 
ences can be made concerning the differences in reading 
ability of two groups of pupils unless the differences between 
their average scores are large. It appears that a scorer is 
not always consistent with reference to his constant errors. 
In Table VIII, Scorer Y and Scorer K show negative differ- 
ences for two sets of papers, and a positive difference for a 
third set. This reversal of policy may be due, in part, to the 
differences in the character of the reproductions; but, doubt- 
less, the unstability of subjective judgment is also a factor. 

In the column headed “‘7;2.” the coefficient of correlation 
between the two sets of scores is given. In the next column 
the probable error of estimate of the obtained scores from 
the corresponding true scores is given. This was calculated 
by the formula, ! 


P.E ry = 6745 o V1 —T2 


This probable error of estimate should be interpreted as a de- 
scription of the amount of variable error, or departure of the 
assigned scores from the true score after the constant error 
has been eliminated. We may, therefore, speak of the prob- 
able error of estimate in this case as the probable variable 
error of the scoring. A probable variable error of scoring of 
3.4 means that, in general, the variable error for the two 
scorers from whom the data were obtained is greater than 
3.4 for 50 per cent of the scores. It is also true that for 50 
per cent of the scores the variable error is less than 3.4. The 
significance of the magnitude of this measure of the variable 
error of scoring becomes more apparent when it is compared 
with the average score. In this table a probable variable 
error of scoring of 9.2 does not necessarily mean that the 
magnitude of the errors was relatively larger than for another 
1 See page 214 for an explanation of this formula. 
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test which has a probable variable error of scoring of 4.8. 
The difference may be due to the size of the units and the 
absolute magnitude of the scores yielded by the two tests. 


2. Reliability 

Obtained scores and true scores. Two applications of 
the same test, or duplicate forms of it, to the same pupils 
will yield many pairs of scores which are not equal, even 
after corrections have been made for practice effect or other 
constant errors. Any lack of objectivity will tend to pro- 
duce differences in the resulting measures. There are, how- 
ever, additional factors which also contribute to these differ- 
ences. As we have shown, it is characteristic of children to 
be variable in their performances. Some children are more 
variable than others, and generally a child will give a more 
variable performance in the earlier stages of his learning 
than later. Many of the causes contributing to these varia- , 
tions are subtle. Some can be partially controlled and kept , 
"constant by appropriate explanation of the nature of the ex- 
ercises to the pupils and by detailed directions concerning 
the method of doing them. The more subtle causes, such 
as the pupil’s attitude toward the test, his emotional state, 
his physiological condition, and his preceding mental ac- 
tivity, cannot be controlled except very slightly. 

A pupil’s true score is defined as the average of a large num- 
ber of measurements by the same test, or by duplicate forms 
of a test, the conditions of measurement remaining constant.! 
This is in accord with our method of obtaining true measure- 
ments of physical objects. For example, in measuring the 
diameter of a spherical ball different results will be obtained, 
and the true measure is the average of a large number of 


1 True scores cannot be obtained because the testing conditions cannot 
be kept constant in the case of repeated measurement. However, the con- 
cept of a true score will be helpful in our consideration of reliability. 
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measures. A strict interpretation is that the average is the 
most likely true measure. The reliability of a test refers to 
the magnitude of the differences between the obtained scores 
and the true scores. These differences are the variable errors 
of measurement. 

The methods of determining the reliability of a test are 
methods of describing the magnitude of these errors. In 
these statements it is assumed that the obtained scores in- 
volve only variable errors. Constant errors may be intro- 
duced for certain groups of pupils, due to the variation of 
certain testing conditions, such as timing. These errors 
tend to become variable for a large group which involves 
different examiners and the administration of the test at 
different times. Practice effect and acquaintance with the 
test produce constant errors. The following discussion of 
methods of determining reliability apply only to the variable 
errors of measurement. Constant errors have been consid- 
ered in Chapter VIII in our discussion of norms. 

Methods of determining the reliability of a test. There 
are five main methods used in determining the reliability of 
a test, and these will now be described. They are: 

(a) Coefficient of reliability. The degree of reliability of 
a set of scores may be expressed by the coefficient of corre- 
lation between them and the scores obtained from a second 
application of the test, or the application of a duplicate 
form of the test.1_ This is called the reliability coefficient 
(ri2). The Pearson formula 2 is recommended, 


Sry 


ty 
aaa Oz Ty 

1 In correlating the two sets of scores it is unnecessary to have them re- 
duced to the same scale. Constant errors do not affect the coefficient of 
correlation. 

? See Chapter x1, for the explanation of this formula, and the method 
of calculating the value of ry2. 
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The meaning of the reliability coefficient (7,2) is indicated 
in the following illustration. Forms 1 and 2 of the Courtis 
Standard Research Tests in Arithmetic, Series B, were 
given to 81 sixth-grade pupils. The correlation table for 
the scores yielded by the addition test is shown in Table IX. 
It is to be read: Four pupils made a score of four on Form 
1; on Form 2, two of these had a score of four, one a score of 
five, and one a score of six. From this table the coefficient 
of correlation was calculated according to the Pearson for- 
mula. This was found to be 71, = .87 £.02. This number 
indicates the degree of general relation between the num- 
ber of examples attempted on Form 1 and the number at- 
tempted on Form 2 by the various pupils. The table shows 
some attempted more examples on the second form than on 
the first. Others attempted more on the first than on the 
second. A few attempted the same number on both forms. 
If all pupils had attempted exactly the same number of ex- 
amples on both forms the coefficient of correlation would 
have been 1.00. However, they did not do this, and this 
fact is indicated by the coefficient of correlation being less 
than 1.00. The amount of difference between .87 and 1.00 
indicates, in a general way, the degree of failure of the pupils 
to make the same score on the two applications of the test. 
The reliability coefficient merely expresses the degree of 
correlation which one may expect between any two sets of 
obtained scores, neither one of which may be considered to 
be true measures. 

The range of talent — i.e., the range of the scores of pupils 
— on which the reliability coefficient is calculated, affects its 
magnitude. If all the pupils belong to a single grade or half 
grade it will be smaller than if the pupils were selected from 
a sequence of two or more grades.! For this reason it is nec- 

1 Monroe, W. S. “Effect of Grade Distribution upon Coefficient of 


Correlation”; in Journal of Educational Research, vol. 11, p. 776 (November, 
1920). This reference gives an extreme illustration of the range of talent. 
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essary to state the range of talent, and to consider it in inter- 
preting the reliability coefficient. The range of talent is 
roughly indicated by the range of school grades from which 
the scores were secured. When the reliability coefficient is 
calculated from scores obtained from pupils belonging to a 
single grade its value is, generally, between .60 and .80. Ina 
few instances, reliability coefficients above .90 have been 
obtained. See Chapter XIII for additional suggestions con- 
cerning the interpretation of coefficients of correlation. 

Relation of the coefficient of reliability to the length of a 
test. Brown’s ! formula expresses the relation between the 
length of a test and its coefficient of reliability. If riz is the 
coefficient of reliability of the test and 7, the coefficient of 
reliability of a test n times as long, 


= nN Tie 
1+(n-1) re 


In this formula, n may be fractional or less than one, as well 
as integral and greater than one. Since rj is always less 
than one, 7, will be greater than rj. Hence, other things 
remaining equal, increasing the length of a test increases its 
reliability. By means of Brown’s formula it is possible to de- 
termine the length of a test necessary to yield any desired 
coefficient of reliability when the coefficient of reliability is 
known for a given length of this test. It is only necessary to 
solve this formula for n, 


Tn 


_ Th (1 — Tr) 


Te (1 — Tp) 
To find the required value of n, substitute the given values 


of 7, and rjg in this equation. 
Calculation of coefficient of reliability from one application 


1 Brown, Wm. Essentials of Mental Measurement, pp. 101-02. Cam- 
bridge, 1911, 
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of a test. When there has been only one application of a 
test we may secure two independent measures by comput- 
ing two scores from the alternate exercises of the test. One 
score may be computed from the performance on the odd 
numbered exercises, and a second score from the even num- 
bered ones. If we represent the coefficient of correlation 
between the scores yielded by the half tests by 7,, we may 
calculate the coefficient of reliability of the whole test by 
means of the formula, 


This will be recognized as Brown’s formula when n = 2. If 
the length of the half tests is multiplied by 2, we have the 
original test. Hence, rj2 is the reliability coefficient of the 
whole test. This method assumes that the test has been con- 
structed so that these parts constitute approximately equiv- 
alent tests. Since the two performances were secured at the 
same time, the testing conditions are probably more com- 
pletely controlled than when the scores are secured from 
two independent applications of the test. Consequently, 
coefficients of reliability calculated in this way will not pos- 
sess the same significance as when they are calculated from 
scores obtained from two independent applications of the 
test. Furthermore, the method cannot be universally ap- 
plied. 

(b) The index of reliability. From the reliability co- 
efficient it is possible to obtain the coefficient of correlation 
between a given set of obtained scores and the corresponding 
true scores. This coefficient of correlation is called the in- 
dex of reliability. If we use rig to represent the coefficient of 
reliability and ry, to represent the index of reliability, the 
formula! is ry,=~/7rg. Since 72 is less than 1, ry is greater 


1 Kelley, T. L. “A Simplified Method of Using Scaled Data for Pure 
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than r}z. The index of reliability, 7,,is the greatest pos- 
sible value of the coefficient of reliability, 712.1 The index 
of reliability expresses the relation between a set of ob- 
tained scores and the corresponding true scores. This is a 
more useful measure of reliability than the coefficient of re- 
liability. Since the calculation of the index of reliability 
requires only the extraction of a square root, it should be 
used in preference to the coefficient of reliability. In inter- 
preting it, the bases of comparison will be correspondingly 
high. Most indices will be between .75 and .90 instead of 
between .60 and .80. 

(c) Probable error of measurement (P.E.y,). In the case 
of an instrument for the measurement of physical objects, 
as, for example, a vernier calliper, repeated measurements of 
a given object will differ slightly, which means that most of 
them differ from the true or average measure. The median 
of these errors or deviations from the true measure is called 
the probable error (P.E.). It is the amount of error which 
is exceeded in just fifty per cent of the measurements. With 
uniform care in the use of the instrument, the P.E. will be 
approximately the same for all objects of the same kind and 
of approximately the same magnitude. We are, therefore, 
able to associate the P.E. with the instrument for the meas- 
urement of this class of objects. In doing this it is implied 
that the instrument is always used under standard condi- 
tions. In the measurement of mental abilities a large num- 
ber of applications of a test are impossible, because the act 
of measurement changes the ability. Instead of a large num- 
ber of measurements of the ability of one pupil, two or three 
measurements of the abilities of each of a large number of 


poses of Testing”; in School and Society, vol. tv, pp. 34, 71 (July 18 and 25, 


1916). 
1 Any calculated nz is subject to a probable error, P.E., and this must 


be understood in the above statement. 
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pupils must be used as the basis for calculating the probable 
error of measurement (P.E.y). Two formule for calculat- 
ing this measure of the variable error will be expiained. 

Calculating the P.Z.,, of a test: Formula A. When two 
forms of a test are given to a pupil the obtained scores, S’ 
and S”, will differ from the true score, T. Thus, S’= T+ e’ 
and S” = T + e”, in which S’ and e’ refer to the score ob- 
tained from the application of Form 1 of the test, and S’”’ and 
e’’ refer to the score obtained from Form 2.1 The differ- 
ences are the variable errors of measurement.? Any set of 
e’s for an unselected group of pupils will form a normal dis- 
tribution whose average is zero, provided the S’s form a nor- 
mal distribution. Thus, the e’s are simply deviations from 
this average. The median deviation from this average is 
represented by P.E.,. When we have two sets of measures 
for the same group of pupils, obtained from two forms of the 
same test, we may use P.E.,, to represent the median devia- 
tion of the errors of the scores obtained from Form 1, and 
P.E to represent the same for Form 2. If the two forms 
are equivalent, and this is included in our hypothesis, P.E.,. 
=P.E.9”=P.E.y of the test. 

Let S’—S”=D,. Some of these differences will be 
positive and some negative and a few exactly zero. They 
will, also, form a normal distribution with the average at 
zero if the original measures are distributed normally. 

It is now necessary to assume that the distribution of 
differences obtained from a group of pupils is equivalent to 
the distribution of differences which would have been ob- 
tained from a large number of pairs of measurements of the 
same ability of one pupil. This assumption, however, is im- 


1 It is assumed that S’ and S” are expressed in comparable units, and from 
the same zero point. If they are not they must be reduced to this basis. 
The method of doing this will be given later. This eliminates constant errors, 

2 We are dealing here only with the variable errors. The constant errors 
will be considered later. 
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plied in our purpose, which is to determine the P.E.y of the 
test for all pupils, at least for all pupils of a given school 
grade. Although the actual P.E.y may be different for 
different pupils and for scores of different magnitudes, as a 
practical device we want a P.E.y which may be used for 
the test with all pupils of a given grade even if this P.E.y 
is only approximate. 

If e’ is subtracted from e” the differences will form a 
normal distribution with the average at zero. The median 
deviation of this distribution is represented by P.E.p,. 
The relation between the three P.E.’s is expressed by this 
equation: 1 


PE.» = P-E.y + P.-E. — 22 P.E.y P.-E. 


In this equation, 7; is the coefficient of correlation between 
e’ and e’’. Since e’ and e” are assumed to be independently 
subject to chance variations, r}2=0. Hence, 


PE», =P.E. + P.Eny 


Since S’= T+e’ and S”’=T+e"’, S’—S”=e'—e’”, and 
hence D,= D,. Thus, the median deviation of the difference 
of the scores, P.E.p,= P.E.p, because the differences are 
identical. Thus, 


PE, =PE4+P.£.y 


If P.E.,» and P.E..» are known, the P.E.p, can be ob- 
tained, but in the present case both P.E., and P.E..» are 
unknown and cannot be found directly. P.E.p, is easily 
found. Thus, remembering that P.E.,= P.E.”= P.E.y 
of the test, 


1 Thorndike, E. L. An Introduction to the Theory of Mental and Social 
Measurements, pp. 190-93. 
Yule, G, Udny. An Introduction to the Theory of Statistics, p. 211. 
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PE.,, =2 P.E-y 
P.E.,,= P.E.y V2 


P.E.p, 
V2 
P.E.p, can be calculated directly from the distribution of 
differences of the scores, but it is simpler to obtain it from 
the average of the differences taken without regard to sign. 
Since the differences are also deviations from their average, 
zero, and the distribution of the differences is normal, the 
usual relations between measures of deviation, average de- 
viation (A.D.), standard deviation (7), and median devia- 
tion (P.E.), hold for the differences of the scores. 

Median difference P.E. = .67450. 

Average difference Av.Dif. = .79790. 
Therefore, P.E. = .8453 Av. Dif. This means, simply, that 
the amount of difference between the two sets of measures 
which is exceeded in 50 per cent of the cases is equal to .8453 
Av. Dif. This formula makes it easy to obtain the P.E.p, 
since the Av. Dif. is the sum of the differences, without re- 
gard to sign, divided by the number of cases. Substituting 
for P.E.p, in the above equation, 


or P.E.y= 


845 
P.E.y = a Av. Dif. of scores. 
P.E.y = 5978 Av. Dif. of scores. (A) 


Application of method. This method may be illustrated 
by its application to the scores obtained from 88 fifth-grade 
pupils when they were given two forms of the addition test 
of the Courtis Standard Research Tests, Series B. The 
score of Form 2 was subtracted from that of Form 1. 
These differences gave the following distribution. The 
central tendency of this distribution is not 0, but is about 
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—.6. This is due to the fact that the scores on Form 2 
were not reduced to the same scale as Form 2 before the 
subtraction was made. For this reason, the result is slightly 
in error, the determination of the P.E.y of the test being 
slightly too large. A  P.E.y of .917 means that half of the 
scores will be in error not more than .917 of an example. 


Difference ~ Frequency 
4 | 
3 4 
g. 4 
1 16 
0; 17 
-1 21' 
-2 12 
-—3 8 
—4 1 
—5 8 
—6 
-7 1 
Total 88 
Av. Dif. = 1.534. 
P.E.y = .6978 X 1.534 = .917. 


Reducing one set of scores to the scale of another set. 
Unless the two sets of measures are expressed in terms of the 
same unit and with reference to the same zero point, they 
are not comparable. It is, therefore, necessary to reduce 
the scores of one test to the scale of the other, or both to a 
common scale. A common method! is to express each score 
as a deviation from the average in terms of the standard 
deviation as a unit. This method also eliminates any con- 
stant error between the two sets of scores. The method, how- 
ever, introduces negative scores, which is undesirable in 

1 Woodworth, R. S. “Combining the Results of Several Tests: A 
Study in Statistical Method”’; in Psychological Review, vol. x1x, pp. 97-123 


(March, 1912). : 
Kelly, T. L. “Comparable Measures”; in Journal of Educational Psy- 


chology, vol. v, pp. 589-95 (December, 1914). 
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computing the P.E.y of a test. They may be avoided 
by the following modification of the method. 

Let S, be the obtained score on Form 1, and Sz the ob- 
tained score on the second form of the test, and S; a score 
equivalent to S, but on the scale of the first form of the test. 
S; and S, being known, it is desired to find S;}. Let A, 
and 0; be, respectively, the average and standard deviation 
of the distribution of the scores obtained from Form 1 of the 
test. Let Ag and 2 represent the same functions of the 
distribution of scores obtained from Form 2. Since Sj is 
on the scale of Form 1 it will have the same average and 
the same standard deviation as S;. Expressing both S; 
and S2 as a deviation from their respective averages 
in terms of the standard deviations as units, we have 
Si =e A, Se me Ag 

and 
1 Ge 
equal, by definition, when expressed in this form, 
Si— A, _Se— Ae 


o7 G2 


Si %2- Ay og = Soo — Ag oy 


Since the two scores Sz and Sj are 


; o o 
8, = 8, 5 + (Ai As a 


This equation gives the relation between the obtained 
scores on Form 2, (S2) and the equivalent score (S;) on the 
scale of Form 1. Thus, if it is desired to reduce Sz scores to 
equivalent scores on the scale of S,, or Form 1, it can be 
done by means of this formula. 

This equation is really the regression equation of Son Se: 


, O71 
Si =S8¢ (re au +K 


The coefficient of correlation, ry2, does not appear in the 
above equation, but the assumption that S, and Se are 
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equivalent scores, but on different scales, implies perfect 
correlation. Hence, r;.= 1. 

Otis ! uses a graphical method for translating the scores on 
one scale into equivalent scores on another scale. The two 
sets of scores are plotted with reference to codrdinate axes, 
and a line of relation is drawn through the point represented 


by the average of the two sets of scores and with the slope 
A.D 


of A De Using the same symbolism as above, the equa- 
tion of the line of relation is 
; A.D. 
Si— Ay= 757 (Ss— Ad 


Since A.D. = .7979 o, if a normal distribution is assumed, 
this equation is equivalent to the equation above. Otis uses 
this line as a “line relation,” and scores on one scale are 
translated into equivalent scores on the other by taking the 
other codrdinate of a point on the line as the corresponding 
score on the other scale. Although Otis does this graphically, 
it amounts to using the formula derived above. 

If the two sets of measures are reduced to a common 
scale by means of the formula 


Sere! Pay aay feged 
81 = S37 +(A; re 


the averages and standard deviations of the two groups of 
scores become equal. This means that we have secured the 
elimination of the effect of those factors which tend either to 
increase or decrease the average of the scores obtained from 
the second application of a test, or from an equivalent form, 
and, also, of those factors which tend either to increase or 
decrease the standard deviation of the scores. Among the 


1Qtis, A.S. ‘‘An Absolute Point Scale for the Group Measurement of 
Intelligence”; in Journal of Educational Psychology, May and June, 1918. 
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factors which affect the average —i.e., produce constant 
errors — are the size of the units, the location of the zero 
point, practice effect, acquaintance with the test, the manner 
of the examiner, and non-equivalence of the tests. The prac- 
tice effect and acquaintance with the test probably tend to 
increase the scores of the bright pupils more than those of 
the less capable. In so doing, the standard deviation is in- 
creased. 

Calculating the P.E.,,of atest: Formula B. The P.E.y 
of a test may be obtained from the coefficient of reliability 
and the standard deviation of the distribution of scores from 
which the coefficient of reliability is calculated. The prob- 
able error of estimate (P.E.z%) is given by the equation! 


P.E..ba4,, = 6745 0) V1- 72, 


In this expression, ¢, is the standard deviation of the dis- 
tribution of scores which is equal to the standard deviation 
of the S” scores, if the two sets of scores are comparable, 
and rj is the coefficient of correlation between the scores 
derived from the two forms of the test. The probable error 
of estimate is a measure of the extent to which Form 1 scores 
depart from perfect correlation with the corresponding Form 
2 scores. We may also obtain a probable error of estimate of 
the departure of the obtained scores from the corresponding 
true scores. If we represent this by P.E. Est,, 


P.E..pu,, = 6745 0, V1—7%, 
Since ry; = Vii 
P.E. pu, = 6745 0; V1— 9 
P.E.ry,, is the same as P.E.x 
Hence, P.E.y = .6745 0, V1— rie (B) 


‘See Chapter XIII, for a more detailed explanation of the probable 
error of estimate, 
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This formula is simpler to apply than formula A (page 210). 
Hence, it is recommended for use. Ifthe distributions of 
scores obtained from the two forms of the test do not yield 
equal standard deviations (cj = og), it is advisable to calcu- 
late both 


o,V1—rie and o2V1—Ti2 


or, what is simpler, use the average of 7, and 2. 

Probable error of measurement expressed as a ratio. The 
magnitude of the P.E.y of measurement becomes signifi- 
cant only when it is compared with the magnitude of the 
measures from which it was calculated. A P.E.y of 1.5 
units for a score of 100 units represents a much higher degree 
of reliability than a P.E.y of 1.5 for a score of 10 units. 
In the former case, the P.E.y is only 1.5 per cent of the 
score; in the latter, it is 15 per cent of the score. Thus, in 
interpreting the P.E.y which is obtained for a test it is 
necessary to compare it with the magnitude of the scores 
from which it was calculated. A simple method of making 


This 


Evy 
Av. * 
expresses the P.E.y as a per cent of the average score of 
the group of pupils. 

Relation of P.E. y, to the length of a test. If the length 
of a test is multiplied by four, so that, other conditions of 
the test remaining the same, the absolute magnitude of a 
pupil’s score will be just four times what it was, the P.E.y 
of the score is affected in its relation to the absolute magni- 
tude of the score in the same way as the P.E. of an average 
is affected by multiplying the number of cases upon which it 
depends by four. The formula for the P.E. of an average is 


o distribution 


Vn 


this comparison is by means of the ratio, 


PP. = 6745 
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In this formula, P.E.4, is the median deviation of similar 
averages from the true average, o distribution is the stand- 
ard deviation of the distribution from which the obtained 
average is calculated, and n is the number of cases from 
which the obtained average is calculated.!_ Since the ab- 
solute magnitude of the standard deviation of the distribu- 
tion remains approximately constant as n is increased, P.E. 4p, 
decreases inversely as »/n. If n is multiplied by 4, P.E. 4», 
becomes one half of its value for n. 

In the case of the P.E.y of a test which is quadrupled in 
length,? its P.E.y = P.E.y original test X 1/4. The ratios 


1 Thorndike, E. L. An Introduction to the Theory of Mental and Social 
Measurements, pp. 188-90. 

2 This statement is based upon several assumptions. In the first place, a 
score is assumed to be an average, or, more strictly speaking, a constant 
times an average. This assumption is not contrary to the observed facts. 
A pupil’s performance may be considered to consist of elements or divisions. 
For example, in an addition test the perception of each of the number sym- 
bols, forming each association, and writing each figure of the sums may be 
considered elements. The pupil’s performance is not constant for all of 
these elements, either with respect to rate or accuracy. If it were possible 
to isolate these elements and to describe each separately, the pupil’s score 
would be the sum of these descriptions, which is nothing but the aver- 


age multiplied by the number of elements. Thus, P.EM = .6745 a 
! x 


In this equation, K isthe number of elements in the test and ox is the stand- 
ard deviation of K elements. 

It is nownecessary to assume that 0 x is the true G; i.e., its magnitude does 
not change as K is increased, or, perhaps more properly, the changes in its 
magnitude as K is increased are so small as to be negligible. This assump- 
tion is always made in applying the formula 


o distribution 

Vn 
to averages. Hence, this assumption is in accord with an accepted practice. 
Upon the basis of these two assumptions we may write 


P.E. Av, = 6745 


P.E.M, = 6745 7 


aoe a er = 


C 
vnk VnK 


PEM = 8745 
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of each P.E.y to the score with which it is associated are 
_ P.E.y quadrupled test . P.E.y original test 
Score of quadrupled test Score of original test 
But, by hypothesis, 
Score of quadrupled test = 4 X score of original test. 
Thus, 
P.E.y quadrupled test _ _P.E.u quadrupled test 
Score of quadrupled test 4 X score of original test 


_ P.E.y of original test X V4 
4 score of original test 


P.E.y quadrupled test _ , P.E.y of original test 


Score of quadrupled test * Score of original test 


This equation means that by quadrupling the length of 
the test its accuracy or reliability has doubled; i.e., the ratio 
of the magnitude of its P.E.y to the magnitude of the 
scores has been reduced to one half of the original ratio. It 
must, however, be remembered, this it is assumed, that the 
original test can be increased in length without producing 


P.EM £: VnK OK 


Dividing one equation by the other, PE, re oor Vn 
P.E.M 
Whenee, P.E.u. = a 
nN 


But, instead of using the average of nK elements as the score, we use n 
times this average when we increase the length of the test n times. 


P.E.M, 497 PEM,» 


n (nK) 
Hence, multiplying the above equation by n, 
nX P.E.M, 


nX P.E£M,. = it 


P.E.M 


sere vn X P.EM x 
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any change in testing conditions. It is only when this 
assumption is realized that the above statement is true. In 
actual practice the assumption will frequently be only par- 
tially realized and hence this relation between the P.E.y 
and the length of a test will be only approximated. 

The P.E. of class scores. The discussion up to this point 
has had to do entirely with the reliability of individual scores. 
If the average is used as the class score, the formula for the 
P.E.y of an average applies 


o distribution of scores 
Vn 


By somewhat the same reasoning as given in the preceding 
section, it may be shown that 


P.E. 49, = 6745 


P.E. individual score 

Vn 
In this formula, n is the number of scores on which the aver- 
age is based. If the distribution of the scores is normal the 
median equals the average. Hence, a similar formula may 
be stated for the median. 

(d) Reliability in terms of the coefficient of correspond- 
ence. The coefficient of correspondence is the per cent of 
the pupils who maintain the same relative position within 
the group on the second form of the test that they had on 
the first form. The method of calculation depends upon 
the definition given to “maintain the same relative posi- 
tion.” One method would be to transmute the scores ob- 
tained into ranks. The per cent of pupils who received the 
same rank in both forms of the test would be the coefficient 
of correspondence. A more satisfactory method is to express 
each score as a deviation from its average in terms of some 
measure of deviation of the distribution of scores. The dif- 
ference between a pupil’s two scores, when expressed in this 


P.E. 49. = 
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form, will be an index of the extent to which he has main- 
tained his “relative position,” and we may arbitrarily define 
“maintaining the same relative position”? to mean that this 
difference is not greater than a certain amount. One writer ! 
has arbitrarily defined “maintaining the same relative posi- 
tion” to mean that the difference between a pupil’s two 
scores shall not be greater than + 1.00 A.D. 

(e) Overlapping of successive grade groups. It is a 
well-known fact that groups of pupils from successive school 
grades exhibit a considerable amount of overlapping in abil- 
ity. One cause of this overlapping is the unreliability of 
the measures. In general, the greater the overlapping the 
greater the unreliability is. Hence, if two tests measuring 
the same abilities have been given to two groups of pupils 
known to differ in ability, the amount of overlapping of the 
resulting scores will afford a rough indication of the relative 
reliability of the two tests. This method is not recommended 
for use. 


8. Discrimination 

Certain criteria of discrimination. This index of validity 
refers to the differentiation of scores for pupils possessing 
different degrees of ability. A measuring instrument lacks 
discrimination when it fails to yield different scores for pupils 
or groups of pupils who possess different degrees of the abil- 
ity being measured. It is obvious that any lack of objec- 
tivity or reliability will result in a lack of discrimination for 
certain pupils. Therefore, the measures of objectivity and 
reliability considered in the two previous sections are, in- 
directly, indices of discrimination. There are, however, cer- 
tain direct criteria of discrimination which may be applied. 

In general, unselected groups of pupils from successive 


1 Courtis, S. A. Measurement of Classroom Products, p. 481. Report of 
Gary School Survey, 1919. - 


~ 
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school grades exhibit an increase in average ability from 
grade to grade. Therefore, a test which fails to show an 
increase in average ability from grade to grade, or which 
shows a very small increase, is likely to be lacking in dis- 
crimination. Other things being equal, that test is the best 
which shows the greatest discrimination between successive 
grade groups. Similar statements can be made with refer- 
ence to chronological-age groups, or other groups which are 
known to differ in the ability measured. 

If the distribution of scores obtained from an unselected 
group of several hundred pupils shows a marked departure 
from the normal distribution, it is evidence of a lack of dis- 
crimination for certain pupils of the group. If a consider- 
able number of pupils make perfect scores we have evidence 
that, for such pupils, the test lacks discrimination. Simi- 
larly, a large number of zero scores is an indication of the 
lack of discrimination for the pupils making these scores. 

Objections to use of a large unit. The use of a large unit 
which cannot be subdivided will prevent proper discrimina- 
tion for many pupils. The units into which instruments for 
the measurement of physical objects are divided are subject 
to subdivision so that relatively small differences may be 
expressed, either in terms of the unit or in terms of a fraction 
of it. Many of our educational tests are of such a nature 
that fractions of units cannot be used in expressing the scores 
of individual pupils. For this reason, if the unit used is 
relatively large, a large number of pupils may receive the 
same score even though they possess different degrees of the 
ability being measured. The degree of precision with 
which measurements of physical objects are made and ex- 
pressed is not justified in the case of mental measurements, 
for the reason that we are unable to make mental measure- 
ments with the same degree of accuracy. For example, 
in the case of the Courtis Standard Research Tests in Arith- 
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metic, Series B, it would be absurd, if possible, to express - 
the measure of a pupil’s rate of work on the addition test 
to the hundredth part of a unit. The probable error of 
measurement for this test is nearly one unit. 

Probably the best procedure to follow in studying the size 
of unit is to ascertain the number of groups into which pupils 
belonging to the same school grade are divided, on the basis 
of their scores. If the number of groups is less than ten, 
when the number of cases is fifty or more, the fact should, 
in general, be accepted as evidence of a lack of proper dis- 
crimination, due to the coarseness of the unit used. On the 
other hand, one should bear in mind that the division of the 
scores into ten or more intervals cannot be accepted as evi- 
dence of valid discrimination. A test may fail to discrimi- 
nate with reference to a specified ability because it measures 
another trait, or because the scores involve errors. 


4. Comparison with criterion measures 

Teachers’ marks. The comparison of scores with teach- 
ers’ marks or ratings as a method of determining the degree 
of validity of a test is based upon two assumptions: (1) 
that the teachers’ marks are measures of the same ability 
that the test measures; and (2) that the marks are more accu- 
rate indices of the ability than the scores yielded by the test. 
Studies of teachers’ marks have shown that, usually, they 
are conspicuously lacking in accuracy, and hence in reliabil- 
ity as general measures. Thus, this method should be used 
with caution. It is most helpful in the case of an achieve- 
ment test which is so general that it covers the entire field of 
a school subject, or in the case of general intelligence tests. 

Since the virtue of the method depends upon the reliabil- 
ity of the ratings given by the teacher, care should be taken 
to secure ratings which will have the highest possible reliabil- 
ity. This is probably best accomplished by some systematic 
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plan of rating, such as the officers’ rating scale which was 
used inthe army. A rating scale of this type for silent read- 
ing is given below.! This scale was used in a junior high 
school where there were four teachers who were acquainted 
with each pupil. The average of the two middle ratings was 
taken as the best criterion of the pupil’s ability to read si- 
lently. A rating secured in this way should afford a highly 
reliable criterion for judging the validity of a general test. 


RATING SCALE — SILENT READING 


Silent reading is defined as ability to comprehend readily and 
accurately, in reading text-books, magazines, and general literature. 

Write on first line (15 value) the name of that pupil, of all you 
have ever known in the grade you are rating, whom you consider 
the best reader; similarly, put, on the last line, the poorest reader. 
Then choose a middle case for the middle line; finally, fill in cases 
half way between the middle and each extreme. 

Now, rate your class as to ability in reading, using the above 
scale and reference cases. (It is urged that, in rating, the inter- 
mediate points, as well as the five reference values given on the 
scale, be used. The class distribution should, in fact, show more 
fours than threes, and more fives than fours.) 


Scale 
WN icpehe nein tre serge oia'> ie! «Sle Garg, sceceIh aie eunierarere Gia malate tear terenere : 
DS xr cteraldia siswaiais a aisie; s.siekeie sit pein .auevts eace onejeverenkenterenete 
Qedeee maine esas oo cere celta teeter caterers Siee Siotererte 
Gavontaesereefs.e slee'e Brarerere ore le ete lets’ sreleterare Paterna terete : 
Dans tia erst ewe Deve Urea silat’ a. anctale Wloletah phe eparenePneerareRaene 
Pupil number Name Rating 
DS Reta tas BREN a CaN ICRA, COE GS areas (a 
Bi eRe ahy tae URN Aas he PII AC BANG Shel a en fay 
Br iuws ig hetelens UE RRR, acs loc ane een Ge) 
BS cs ba saan Sands AR RVS R el s eeech 9 aOR beg 


1 Pressey, S. L. and Pressey, L. W. “The Relative Value of Rate and 
Comprehension Scores in Monroe’s Silent Reading Test as Measures of 
Reading Ability”; in School and Society, vol. x, pp, 747-49 (June 19, 1920). 
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The validity of prognostic tests. In the use of school 
marks, as a criterion of validity, it is necessary to recognize 
two types of tests with reference to function. Certain tests 
are designed to measure the achievements of pupils. These 
achievements are the results of instruction and training, 
provided either by the school or by other educational agen- 
cies. Other tests are intended to measure a pupil’s capacity 
to achieve within a given field. The function of these tests 
may be described as prognostic. Achievement tests also 
fulfill a prognostic function, because the measure of a pupil’s 
success at any time is an indication of his probable future 
success. Therefore, in a sense, all tests have a prognostic 
function. We are, however, concerned here with those tests 
which are specifically designed for prognostic purposes. 

A pupil’s school marks represent his success. It is true 
that these marks are not always measures of a pupil’s real 
success, but they do represent his actual success as deter- 
mined by the school. Therefore, when studying the validity 
of a prognostic test, comparison with the school marks which 
a pupil receives in the future constitutes a practical basis for 
judging the validity of the test. It may be pointed out that 
the correlation between the scores yielded by a prognostic 
test and the school marks, which represent the pupil’s future 
success, cannot be expected to be perfect. The failure of 
some pupils to utilize their capacity to learn, the unequal 
effectiveness of the instruction which the schools provide 
and the inaccuracy of teachers’ marks all alike combine to 
reduce the degree of correlation. 

Measures yielded by other tests. If two tests are con- 
sidered to have the same function, and one is known to have 
a high degree of validity, comparison of the second test with 
it furnishes an index of the validity of the second test. The 
method of making this comparison is to have the two tests 
given to the same pupils under the same conditions, and to 
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correlate the resulting scores. Except by chance, the co- 
efficient of correlation obtained cannot be greater than the 
coefficient of reliability of the second test. The value of 
this method is dependent upon the availability of tests which 
have a high degree of validity. Since in most fields the va- 
lidity of our tests has not been determined, this method can- 
not be said to have much value. The usefulness of this 
method is also limited by reason of the fact that two tests 
are seldom considered to have identical functions. Fre- 
quently, one reason for the construction of the proposed 
test is that it fulfills a different function. In general, this 
method should be considered to yield only a rough indica- 
tion of the validity of a proposed test. It is, perhaps, most 
useful in the field of general intelligence, where the Stanford 
Revision of the Binet Scale for Measuring Intelligence is 
considered to be highly valid, and has been frequently used 
as the basis of judging the validity of other tests of general 
intelligence. 

Comparison with composite test scores. This method is 
based upon the principle that the average of a number of 
measures is likely to be nearer the true measure than any one 
of the measures used in obtaining the average. The com- 
posite score is the average of the scores of the same pupil 
after they have been reduced toa common scale. It is based 
upon scores obtained from different tests which are designed 
to measure the same ability or group of abilities in the same 
general field. In considering the validity of a test by this 
method it is necessary to distinguish between one whose 
function is general, over a rather large field, and one whose 
function is limited to a very narrow field. In the case of a 
general test, the exercises which the pupils are asked to do 
in one test represent only a relatively small sample selected’ 
from the large field. This would be true in spelling. There- 
fore, the composite score of two or more general tests in the 
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same field should be a more accurate measure of the pupil’s 
ability than the score derived from a single test, because of 
the larger sample on which it is based. In case the exercises 
of a test are different from those which the pupil is accus- 
tomed to, different types of performances will, in general, 
result in composite scores which are more valid than those 
obtained from a single test. For example, in the case of 
silent reading, a composite score based upon a group of tests 
which call for a variety of performances will probably be more 
valid than scores based upon a single type of performance. 
One reason for this is that some pupils will excel in one type 
of reading, and others in another. Since silent reading isa 
complex activity and we may recognize several different 
types of reading, a general measure of a pupil’s reading abil- 
ity is the average or a composite of his ability in all types of 
silent reading. 

Comparison with composite measures is particularly use- 
ful when the measuring instrument is made up of a battery 
of sub-tests, as in the case of most group intelligence tests. 
In the interest of economy it is desirable to reduce the num- 
ber of sub-tests to a minimum. The procedure usually fol- 
lowed is to try out a large number of sub-tests which, taken 
together, are considered to cover rather completely the field 
of general intelligence. The composite of the scores yielded 
by all of these tests is then formed, and the group of sub-tests 
which correlate most highly with this composite is selected 
to form the final measuring instrument. By this procedure 
the sub-tests are selected which best fulfill the function of a 
larger group of tests. Therefore, they are considered to 
yield the most valid measures of intelligence which can be 
obtained by such a number of sub-tests. In selecting the 
sub-tests in such cases it is also customary to give attention 
to the inter-correlation between the various subtests, and 
select those which exhibit a low degree of correlation with 
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each other. The reason for this is that when a high de- 
gree of inter-correlation is exhibited the two tests tend to 
fulfill the same function. 


5. Inferences concerning validity 


These based on structure of test. Under the head of re- 
liability we have considered the accuracy of mental meas- 
urements. However, to show that the measurements made 
by a given instrument possess a high degree of reliability is 
not sufficient evidence to prove that the instrument measures 
the ability which it is considered to measure; i.e., that a con- 
stant functional relation exists between its scores and the 
abilities specified by its function. A test always yields 
some sort of measure of something, but it may happen that 
it measures an entirely different ability, or a combination of 
abilities in which the ability it is intended to measure plays 
@ minor part. It is, therefore, necessary to supplement 
studies of the objectivity, reliability, and discrimination of 
a test by such inferences as can be made concerning its va- 
lidity. The basis for these inferences is the structure of the 
test and the plan of its administration, including the descrip- 
tion of the pupil’s performance. This is supplemented by 
the nature of the ability measured. This is the most com- 
mon method for “sizing up” a test. One examines a test, 
noting certain characteristics, and forms a judgment concern- 
ing its validity. The application of this method is usually 
not systematic or complete. Significant characteristics may 
be overlooked. A detail may be singled out and the test 
condemned or approved on the basis of it alone, whereas a 
complete consideration of the structure of the test and the 
nature of the ability being measured would result in the 
opposite conclusion. 

When a test lacks in validity. The characteristics of a 
test which should be noted, in making inferences concerning 
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the constancy of the relation existing between the scores it 
yields and the abilities specified by its function, were con- 
sidered in Chapter IV, under the head of requirements of 
test construction. We may summarize them here by stat- 
ing the conditions which justify the inference that the test 
is lacking in validity. Our final estimate of the validity of a 
test should be the consensus of information secured under 
the head of objectivity, reliability, discrimination, and com- 
parison with criterion measures, plus such inferences as we 
may be able to make. A test may be considered lacking in 
validity when any of the following conditions can be shown 
to exist: 

1. The exercises of the test call for the functioning of 
abilities other than those specified by its function, provided 
these other abilities cannot be considered approximately 
constant for all pupils. 

2. When a variety of methods of work may be used by 
the pupils, the method of giving the performance is not spec- 
ified in the directions to pupils. 

3. The existence of testing conditions which do not affect 
all pupils alike. 

4, The abilities specified by the function of the test do not 
function throughout the test. 

5. In the case of a general test, the exercises are not rep- 
resentative of the field of ability defined by the function. 

6. Some pupils are not given opportunity to demonstrate 
their ability. 


VI-VIII. Vauiprry or Sieniricance, Norms, AND 
PRACTICAL CONSIDERATIONS 
Validity of significance. The expenditure of time and 
money which is invested in the application of a test is justi- 
fied only when the resulting measures are useful in the school 
activities enumerated in Chapter III. The attainment of a 
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high degree of validity does not prove the worth of the test 
unless its function is in agreement with our educational ob- 
jectives. No matter how accurate or how valid the scores 
yielded by a test may be, if they are not measures of a trait 
which it is worth while to have, the measuring instrument 
must be considered to have little value. For example, a test 
which measures very accurately a pupil’s ability to cross out 
?’s from a printed selection has little value, because the 
crossing out of ?#’s is a trait which has practically no relation 
to our educational objectives. In general, the trait which an 
achievement test measures must be included in our recog- 
nized educational objectives if the test is to be considered a 
valuable measuring instrument. A somewhat similar state- 
ment can be made with reference to prognostic tests. In 
order to be significant, a prognostic test must predict a pupil’s 
success in achieving a worth-while trait. 

A test should yield measures of ability in terms of the di- 
mensions which are most significant with respect to our edu- 
cational objectives. A trait is described in terms of its di- 
mensions of rate, accuracy, and level of difficulty on which it 
functions. These three dimensions are not equally signifi- 
cant in all fields of ability. In some cases, only one or two 
are significant. For example, in the case of spelling, the rate 
is not significant. In solving problems in arithmetic, the 
rate of work is much less significant than the accuracy of the 
reasoning process. The level of difficulty is seldom an im- 
portant dimension; i.e., it is seldom included in our recog- 
nized educational objectives. It is desired that pupils ac- 
quire abilities, not in order to do difficult things, but in order 
to do things which are socially useful. It is true that many 
socially useful things are also difficult, but it is not true that 
the most difficult things are always the most socially useful. 
In fact, this is very obviously not true. For example, in the 


case of spelling, the most difficult words are not the most 
useful words. 
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This requirement may be stated from another point of 
view, in the case of achievement tests. The norms which are 
obtained are usually interpreted, and rightly so, as objec- 
tives or goals to be achieved. Unless these are described in 
terms of significant dimensions the acceptance of the norms 
as goals to be attained will be inconsistent with our recog- 
nized educational objectives. 

Norms. In order to interpret the scores yielded by tests, 
norms are necessary. In the preceding chapter the different 
types of norms were diccussed, and the limitations of each 
were pointed out. In the complete description of a test it is 
necessary to know what types of norms are available. It is 
also imperative that we know the validity of these norms. 
One factor to be considered in this connection is the popu- 
lation groups, and the number of cases on which they are 
based. It goes without saying that the population groups 
should be representative. Certain population groups ap- 
pear to differ sufficiently so that separate norms are re- 
quired. For example, separate norms appear to be required 
for rural schools and for city schools. If a single general 
norm is issued to be used with both groups, the scores on 
which it is based should be obtained from both groups. It is 
also necessary to know at what time during the school year 
the scores on which the norms are based were obtained. The 
directions followed in the giving of the test and in the scoring 
of the test papers is another factor to be considered. A given 
set of norms can be used in interpreting only those measures 
which are obtained by following the same directions. 

The evidence at hand indicates that if pupils are ac- 
quainted with a form of a test they make higher scores. 
Therefore, the effect of acquaintance with the form of test 
needs to be known in connection with the norms. If the 
norms have been derived from scores which represent a first 
application of the test, they are not suitable to use in inter- 
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preting scores obtained from a second application of the test. 
When a test is used the second time it is desirable to have 
a duplicate form. For a number of our measuring instru- 
ments duplicate forms have been provided. They had 
been constructed in such a way that they were expected to be 
equivalent. It has, however, been found that exact equiv- 
alence was not secured in most cases. Therefore, it is nec- 
essary that the degree of equivalence be known, or separate 
norms for the different forms established. 

Practical considerations. The questions to be investi- 
gated under this head need no explanation. They are, how- 
ever, important questions when a test is being considered 
for general use. A test which requires a large expenditure of 
time and whose cost is relatively large may be justified for 
experimental purposes, but it is not appropriate for general 
use. 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1. Why should an outline, such as is given in this chapter, be followed 
in describing an educational test? 

2. What is the value of knowing the reliability of an educational test? 

3. Is it reasonable to expect that some time educational tests may be 
constructed which will yield perfectly reliable scores? 

. What is the meaning of validity of significance? 

. What are the sources of the unreliability of a test? 

. How would you go about increasing the reliability of a test? 

. What are the advantages of expressing the reliability of a test in 
terms of the probable error of measurement (P.E.x)? 

. Why do we experience difficulty in determining the validity of a test? 


eo 2A 
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CHAPTER X 
USING STANDARDIZED EDUCATIONAL TESTS 


Standardized educational tests not teaching devices. Oc- 
casionally, teachers and supervisors have expressed sur- 
prise that pupils did not appear to exhibit any greater degree 
of ability after a standardized educational test had been 
given to them than the pupils possessed before. Such per- 
sons appear to have expected that in some mysterious way 
the pupils would be made different by the giving of the test. 
A standardized educational test is not a teaching device. It 
is not intended to change a pupil’s ability. It is merely an 
instrument which teachers and other school officials may 
use to secure information about the achievements of pupils. 
Whenever such information is needed in carrying on the 
work in the school, or can be used as a basis for increasing 
the efficiency of the school, a standardized educational test 
becomes a means to these ends. 

A few persons appear to have thought of standardized edu- 
cational tests as playthings which might be used for their 
own entertainment, or for the entertainment of an audience 
at teachers’ meetings. Beautiful charts in colored inks have 
been used to represent the scores obtained. Because such 
charts were tangible and new they attracted much attention. 
Standardized educational tests are, however, not playthings, 
neither are they teaching devices. It is only as the infor- 
mation which they yield is used in increasing the efficiency 
of the school that the tests become worth while. We may 
classify and indicate their usefulness under the two main 
headings of (I) Teaching, and (II) Supervision. 
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I, Tue Uses or STANDARDIZED EDUCATIONAL TESTS IN 
TEACHING 


In Chapter III we enumerated several activities of the 
school in which measurements of the abilities of pupils are 
required. It is not imperative that these measurements be 
made by means of standardized educational tests, but in 
many cases more accurate and more definite measurements 
may be secured by using these instruments. In addition 
to the uses of standardized educational tests implied in the 
demands of these activities, these instruments are helpful in 
certain other respects. They are useful in motivating school 
work. The norms established for the tests are useful as 
educational objectives. The results of measurement may 
also be used in a valuable kind of school publicity. 

General considerations concerning the use of standard- 
ized educational tests. The use of standardized educational 
tests involves more than merely giving them, scoring the test 
papers, and tabulating the resulting scores. Before a test 
can be given it must be selected. Since we now have avail- 
able a large number of standardized educational tests, the 
intelligent selection of an appropriate test is frequently not a 
simple matter. After the scores are obtained they must be 
interpreted in terms of the needs of the pupils, and the fu- 
ture school procedure must be planned to meet these needs. 
If undesirable conditions are revealed they must be remedied. 
If a high degree of efficiency is shown to exist the work of the 
school should be directed so that it will be maintained or 
increased. 


1. Selection of standardized educational tests for a specific 
purpose 


Definition of need. In certain of the preceding chapters 
it has been noted that standardized educational tests may 
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differ widely with respect to function, even though their 
titles imply that they have the same function. In Chapter 
III we indicated the general character of the measures re- 
quired by certain activities of the school. For certain pur- 
poses we need rate tests; for others we need power tests. 
For some school activities we require general or average 
measures of the abilities of pupils in a given field; in other 
cases we need separate measures of specific abilities. For 
some purposes we require only average or median measures 
of groups of pupils; for others measures of the abilities of in- 
dividual pupils are required. In selecting a test one should 
first define his need. Then, with this need in mind one 
should carefully inquire into the precise functions of the 
available tests. Only in this way will it be possible to plan 
a testing program which will adequately meet one’s needs. 
If the test is to be administered by the teachers it is highly 
desirable that it be easy to give to the pupils. This is partic- 
ularly important when standardized educational tests are be- 
ing used for the first time in a school system. It should be 
noted that frequently measures of both general intelligence 
and achievement are necessary.! 

Validity. Standardized educational tests are not perfectly 
valid; i.e., they do not accurately measure the abilities spec- 
ified by their function. Some yield measures which involve 
relatively small errors; others yield measures in which the 
errors are large. For this reason it is important that one be 
informed concerning the validity (including reliability) of a 
test when formulating a testing program. Unfortunately, 
as we indicated in Chapter IX, there is no single numerical 
index of the validity of a test. One should, however, ascer- 
tain the index of reliability or, better still, the probable error 
of measurement ? of the tests considered. Unfortunately, 

1 See page 43ff. 
* See page 206ff. 
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this is not always available. When it is not, such informa- 
tion as is available should be secured. 

Norms. It is also very important to inquire concerning 
the nature and validity of the available norms. In Chapter 
VIII, we pointed out that a given set of norms is valid only 
for specified testing conditions. Norms which are appropri- 
ate for first-trial scores are not appropriate for second-trial 
scores, unless appropriate corrections are made. Norms 
which are appropriate for pupils unacquainted with the 
testing procedure are not appropriate for pupils who are 
accustomed to taking standardized tests. In case there is 
more than one form of the test, one needs to inquire con- 
cerning the equivalence of these forms. 

Cost of testing material. Finally, but not least, one must 
consider the cost of the testing materials and the time that 
will be required for their administration. The cheapest test is 
not necessarily the best, neither is the most expensive nec- 
essarily a superior test. In considering the cost of testing 
materials it must be remembered that frequently a test meas- 
ures only one ability. In such cases, if we are to measure 
the achievements of pupils within the field of a school sub- 
ject, it will be necessary to give a battery of tests rather than 
asingle test. For example, the Courtis Standard Research 
Tests, Series B, measure ability only in the field of the oper- 
ations with integers. In order to cover the field of arith- 
metic it would be necessary to add tests for common and 
decimal fractions and for the problem field. Furthermore, 
as we have pointed out in Chapter III, a testing program 
should include a general intelligence test, as well as tests 
which measure achievement. This means that frequently 
a testing program will involve giving not merely one test, 
but several tests. If only one or two tests are to be given 
the per-pupil cost will not be large, even if each test is rela- 
tively expensive. On the other hand, if the testing program 
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requires the giving of a number of tests the per-pupil cost 
will be large unless the testing materials are relatively cheap. 

Time cost of testing and scoring. The purchase price of 
the testing materials is not the only cost. It requires the 
time of the pupils to take the test. It takes the time of the 
teacher or someone else to score the test papers, and to as- 
semble the scores that are obtained. Reasonable expendi- 
tures of money for testing materials and of time in the giv- 
ing and scoring of test papers can be abundantly justified. 
Teachers or other school officials should not look upon edu- 
cational tests as something additional or superfluous to the 
work of the school. Certain activities of the school require 
measurements of the abilities of pupils. If they are not se- 
cured by means of standardized educational tests, these 
measurements must be made in less accurate ways. The 
use of standardized educational tests should be thought of 
as an integral and valuable part of the work of the school. 
The time which a teacher spends in the scoring of test papers 
or in the giving of a test is likely to be just as profitably spent 
as the time used in planning a lesson or in holding arecitation. 
However, the amount of time required should be considered 
with reference to the amount of information which the test 
yields. If atest requires the expenditure of a large number of 
hours of the teacher’s time but yields a large amount of in- 
formation, its time cost per unit of information will be rel- 
atively small. On the other hand, a test which requires 
but little time may be relatively expensive because it yields 
only very meager information. When the teachers of a 
school system are just beginning the use of standardized ob- 
jective tests it is desirable to avoid tests which require a large 
expenditure of time. 

It will seldom be possible to choose a test which is ideal in 
every respect. It will generally be necessary to effect a com- 
promise. It is imperative that the test selected be appro- 
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priate to one’s needs. When there is opportunity for a fur- 
ther choice, no definite rule can be stated. What is the best 
test will depend upon circumstances. 


2. The administration of standardized educational tests. 


Testing by a single teacher. When a teacher undertakes 
a testing program as an individual enterprise he should be- 
come familiar with the directions before attempting to ad- 
minister the test to his pupils. Many of our standardized 
educational tests are accompanied by sufficiently explicit 
directions so that a teacher who is willing to invest an hour 
or two in the study of the test and its directions will have no 
difficulty in understanding exactly how it is to be adminis- 
tered. In some cases detailed directions to examiners have 
not been formulated. When this condition exists it will be 
necessary for the teacher to formulate the details of his pro- 
cedure. In doing this he should bear in mind that the pur- 
pose of giving the test is to secure truthful information con- 
cerning the achievements of his pupils. The information 
will not be truthful unless the testing conditions approxi- 
mate those for which the norms are stated. Therefore, the 
teacher should try to approximate as closely as possible the 
standard testing conditions. Any variation from the stand- 
ard testing conditions, which tends to either increase or de- 
crease the scores, will tend to make the information untruth- 
ful and hence lacking in usefulness. 

Administering a testing program throughout a school sys- 
tem. When a testing program is being given throughout a 
school system it is desirable to have a high degree of uniform- 
ity in the giving of the tests and in the scoring of the test 
papers. Unless the testing conditions approximate uni- 
formity it will not be possible to make valid comparisons 
between classes or other groups of pupils. Different exam- 
iners will tend to vary slightly in the administration of a 
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test. For this reason it has been urged by some that all of 
the testing throughout a city should be done by one person, 
or at most by a small group of persons who have been specif- 
ically trained for this purpose. There is no doubt that such 
a practice will tend to secure a higher degree of uniformity in 
the administration of a test. However, it is probably best 
to have the tests administered in each room by the classroom 
teacher. In the first place the pupils are acquainted with 
the teacher, and he will be more likely to secure more normal 
responses than a stranger, especially in the case of younger 
children. However, the most important reason is that it is 
extremely desirable to have the teacher in sympathy with 
the testing program, and for him to be acquainted with the 
nature of the tests. Ifthe tests are given by anyone else the 
teacher may feel that they are being imposed upon him from 
above, and that the results are unjust measures of his work. 
Unless the tests are given for supervisory purposes the 
teacher is largely responsible for the interpretation of the 
measures in terms of the needs of the pupils. He must also 
plan the details of the remedial procedure and, what is more 
important, he must execute it. The teacher will be better 
qualified to do this if he administers the tests and scores the 
test papers. The loss of uniformity in administration is 
probably more than compensated for by the gains accruing 
from the teacher’s increased acquaintance with the tests. 

It is desirable that a testing program for a school system be 
in charge of one person, or at the most of a small committee. 
In cities where there is a bureau or division of educational 
research the director in charge of the bureau would naturally 
undertake this work. In other places this may be done by 
the city superintendent, or by someone designated by him. 
If the testing is confined to a single building, the building 
principal would be the logical one to take charge. After the 
testing program has been formulated the teachers concerned 
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should be called together. Unless they are familiar with 
the tests to be given it will be helpful to administer the tests 
to them in exactly the same way as if they were a group of 
pupils. This gives the teachers a very definite notion of how 
to proceed in administering the tests to their pupils. In 
many cases the tests should be explained to the teachers, 
particularly their functions. There should be a definite 
attempt to create an attitude on the part of the teachers 
that a test is used to secure truthful information concerning 
the achievements of their pupils. They should be given to 
understand that truthful information cannot be secured 
unless there is close conformity with the directions for ad- 
ministering the test, and that this requires explicit and de- 
tailed compliance with the directions. 

The scoring of test papers. Many of our standardized 
educational tests are constructed so that only one answer is 
correct. In scoring such tests the usual rule is to give no 
credit for answers that are not correct. When a test of this 
type is accompanied by a list of correct answers the scoring 
is simple. It is only necessary for the scorer to compare a 
pupil’s answers with the correct answers, and to compute 
his score. When a pupil is required to give an answer in the 
form of a phrase or sentence, a great variety of performances 
will be secured. It is not possible for the test-maker to an- 
ticipate all of the answers which pupils will give. Even if 
such a test is accompanied by elaborate directions for scoring 
there will still be difficulty in deciding whether some answers 
should be counted as correct or not. In such cases the scor- 
ing is not a simple matter. In order to secure uniformity 
the one in charge of a testing program should call a meeting, 
at which a number of representative papers should be taken 
up and scored by the group as a whole. By doing this sup- 
plementary rules for scoring can be formulated. This will 
tend to increase materially the uniformity of the scoring. 
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Training in the use of a quality scale. As we pointed out 
in Chapter VI, the description of a pupil’s performance in 
terms of a quality scale is subjective. Unless the scorers 
have had appropriate training the scores assigned by them 
will involve large constant errors as well as large variable 
errors. A reasonable amount of practice will materially 
reduce the magnitude of both types of errors. Therefore, in 
the measurement of ability in handwriting, written composi- 
tion, or other subjects where a quality scale is used, the 
teachers should be called together for a period of training in 
the use of the quality scale. Each teacher should be pro- 
vided with a copy of the scale, and the general procedure to 
be followed in using it should be discussed. The teachers 
should then be asked to rate a number of representative 
samples. 

It is desirable that these be samples of known value. In 
case such samples are not available, representative samples 
taken from the pupils’ test papers may be used. The sam- 
ples should be numbered and each teacher should keep a rec- 
ord of the score which he assigns to each paper. These 
should then be assembled in a summary tabulation.! This 
will probably show marked differences in the scores assigned 
to the same paper. The teachers who assigned extreme 
scores probably did so because they gave attention to some 
characteristic that was not considered important by the 
majority of the teachers. They should be cautioned to 
avoid being unduly influenced by such characteristics. The 
averages of the scores assigned by different teachers will 
give some indication of the constant error. One will likely 
find that some teachers in general are inclined to assign 
scores that are too high, others scores that are too low. If 
the true values of the samples are known they should be 
given to the teachers, and each teacher asked to compare 

1See page 197, 
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his scores with the true values. In case the true values of 
the samples are not available, the average of the scores as- 
signed to them by the group may be used as a basis of com- 
parison. A small amount of training will materially de- 
crease the subjectivity of the scoring. 

Scoring by pupils or clerks. The scoring of test papers, 
especially in the case of a comprehensive testing program, 
requires the expenditure of a considerable amount of time. 
When it is desirable that the test scores be reported promptly 
to a central office, this places a considerable burden upon 
the teachers. In order to eliminate the drudgery of scoring 
it has been proposed that the test papers be scored in the 
upper grades of the elementary school, by reading the cor- 
rect answers to the pupils and having them mark the test 
papers. When no questions can arise concerning what an- 
swers are to be accepted as correct, this procedure will gen- 
erally prove satisfactory. ‘The pupils should exchange 
papers before the scoring is begun. A rescoring by other 
pupils will eliminate most of the errors. In some schools the 
pupils in the seventh and eighth grades have been asked to 
score in a similar way the test papers for the lower grades. 
Whenever the scoring is done by pupils, a few papers selected 
at random should be checked by the teacher or some other 
competent person. 

When questions arise concerning what answers should be 
accepted as correct, the scoring cannot be done satisfactorily 
by pupils. In such cases the teacher may be relieved of the 
burden of scoring by employing competent clerks. In some 
cases pupils from teacher-training classes in the high school, 

1 Van Wagenen, M. J. “The Accuracy with which English Themes may 
be Graded with the Use of English Composition Scales”; in School and 
Society, vol. 1x, pp. 441-49 (April, 1921). 

Theisen, W. W. “Improving Teachers’ Estimates of Composition 


Specimens with the Aid of the Trabue Nassau County Scale”; in School 
gnd Society, vol. vu, pp. 143-50 (February, 1918). 
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or from some of the commercial classes, have been drafted 
for this work. If such pupils are selected with reference to 
their accuracy in clerical work, and are given preliminary 
training, the scoring can be done satisfactorily under care- 
ful supervision. 

It should, however, be noted that a teacher who does not 
score the test papers of his pupils misses an opportunity 
for an intimate acquaintance with their mental processes. 
This is particularly true in the case of those tests where it is 
necessary for the pupil to formulate his answers in terms of 
phrases or sentences. Frequently, the form of a pupil’s an- 
swers will reveal much more concerning his instructional 
needs than will his score on the test.. 


8. Planning remedial procedure 

Interpretation of scores. The scores yielded by a stand- 
ardized educational test must be interpreted with reference 
to the use that is to be made of them. A number of uses 
will be discussed separately, but there are certain general 
matters which may be mentioned here. It will generally be 
helpful to assemble the scores in the form of a distribution. 
Most standardized educational tests are accompanied by a 
class record sheet and directions for tabulating the scores. 
This is usually the first step in their interpretation. From 
the distribution of scores or from an arrangement of the test 
papers a class score is calculated.; This is, generally, the 
median, although sometimes the average is used. The me- 
dian or average score is necessary for interpreting the stand- 
ing of the class as a whole. It is also helpful in interpreting 
the scores of individual pupils. 

Types of errors. ‘The measures yielded by standardized 
educational tests involve two types of errors.! The variable 


1 See page 198 and 202, 
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errors of measurement tend to cancel each other in the me- 
dian or average score of a group. For a group of 25 pupils 
the probable variable error of measurement of the class score 
is one fifth as large as it is for the individual pupils. Fora 
group of 100 pupils it is one tenth as large. Many of our 
standardized educational tests were originally designed to 
be used in securing average measures of groups of pupils. 
When used for measuring the abilities of individual pupils 
the variable errors of measurement are relatively large. In 
fact, in all measures of mental traits, the variable errors of 
measurement are much larger than we are accustomed to in 
the measurement of physical objects. Our best means of de- 
scribing these errors is in terms of the limit which is exceeded 
in only a certain per cent (usually 50 per cent) of the scores.! 
This makes it impossible for us to know anything about the 
actual variable error of measurement in a score of a particu- 
lar pupil. We can only say what the chances are that it does 
not exceed a specified limit. For this reason it is necessary 
to exercise a great deal of caution in interpreting the scores 
of individual pupils. 

It is also necessary to take into account the constant er- 
rors. ‘These are produced by some change in the testing 
conditions or by some other factor which affects all pupils 
alike. The norms of a test are stated with reference to cer- 
tain testing conditions. If these conditions are modified in 
any way, or other factors are introduced which tend to in- 
crease or decrease the scores of all pupils, misleading inter- 
pretations are likely to be made. 

Graphic representation of results. In comparing scores 
with appropriate norms it is frequently helpful to represent 
them graphically. This is particularly true when the results 
of testing are being brought to the attention of a group 
of teachers or other persons. Sometimes unexpected con- 

1See page 207. 
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ditions will be revealed when the scores are represented 
graphically." 

Remedial procedure. The scores yielded by standardized 
educational tests should be interpreted in terms of the re- 
medial procedure which the pupils need. For example, if 
the scores are being interpreted with reference to the educa- 
tional guidance of the pupil the interpretation should be 
stated in terms of the school subjects which he should pur- 
sue and should not pursue. This involves more than merely 
saying that a pupil’s score is above, up to, or below the norm. 
It requires that the one making the interpretation be ac- 
quainted not only with the test from which the scores 
were obtained, but also with school procedure. One cannot 
interpret scores in terms of remedial instruction unless he 
is well acquainted with methods of instruction. Neither 
can one interpret scores with reference to the promotion and 
classification of pupils unless he is acquainted with this 
phase of the organization of a school. 

The interpretation of scores and the planning of remedial 
procedure is, perhaps, the most difficult part of the use of 
_ standardized educational tests. It is, however, the most 
important part. Without this step the giving of an educa- 
tional test will have only limited value. When a test has 
been given throughout a school system it will be very help- 
ful to call together all teachers concerned and discuss the in- 
terpretation of the scores obtained. In fact, the teachers 
have more need of assistance in this step of the use of stand- 
ardized educational tests than they have in the administra- 
tion of them. 

The use of standardized educational tests by the super- 
visor and by the teacher. The following discussion of the 

1 Monroe, Walter 8. Measuring the Results of Teaching, chaps. mt 
and 1x. Houghton Mifflin Company, 1918. 


Alexander, Carter. School Statistics and Publicity, chap. x1. Silver 
Burdette and Company, 1918, 
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uses of standardized educational tests is given in terms of 
the persons who use them; i.e., the supervisor! and teacher. 
One disadvantage in this procedure is that both the teacher 
and the supervisor participate in a number of the activities 
of the school. For the purpose of discussion we have arbi- 
trarily divided the activities into two groups, placing in one 
those activities in which the teacher is most prominent, and 
in the other those in which the supervisor is most prominent. 
However, it will be noted that the teacher has a part in a 
number of the activities assigned to the supervisor, and the 
supervisor has part in some of the activities assigned to the 
teacher. 

The teacher’s uses of standardized educational tests. 
The teacher makes use of standardized educational tests in 
carrying on the following activities: 

I. Diagnosis of pupils with respect to achievement for 
the purpose of planning remedial instruction. 

II. Diagnosis of pupils with respect to study procedure 
for the purpose of assisting them in acquiring good methods 
of study. 

III. Setting immediate educational objectives. 

IV. Creating motives for learning. 

V. Reporting the achievements of pupils to parents. 

In Chapter III we indicated the general type of informa- 
tion which the first two of these activities require. Stand- 
ardized educational tests are helpful in setting up educa- 
tional objectives and in creating a motive for learning. 


g. Diagnosis of pupil difficulties by the use of stand- 
ardized tests 
Diagnosis with reference to grade norms. Until recently 
‘the diagnosis of pupils with respect to achievement con- 


1 The supervisor is used here as a general term to include superintendent, 
principal, or special supervisor. 
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sisted merely of comparing the scores with the grade norms, 
and in noting the character of the distribution of the scores. 
If the class was being diagnosed as a whole, a class median 
distinctly below the grade norm was interpreted to mean 
that the class as a whole.needed additional instruction. The 
pupils might need more of the training that they had re- 
ceived, or possibly their need was for a different type of train- 
ing. Sometimes this condition was interpreted to mean 
that the trait shown to be below standard had not received 
sufficient emphasis. In case the class score was above stand- 
ard, or the individual scores were too widely scattered, cer- 
tain inferences were drawn with respect to instructional 
needs of the pupils.! 

In the case of individual pupils a similar procedure was 
followed. Scores below the grade norm were interpreted to 
signify a weakness in the pupil’s training. For some reason 
the pupil had failed to respond in a satisfactory way. He 
needed additional training of the same kind or of a different 
kind. Perhaps the pupil had failed to apply himself, and 
the teacher should take steps to secure a stronger motive.? 

Diagnosis of pupils in terms of achievement quotients. 
The type of diagnosis described above is valuable. A 
teacher who intelligently uses standardized objective tests 
in this way will materially increase his efficiency as an in- 
structor. Such diagnosis, however, fails to take account of 


1 Monroe, W.S. Measuring the Results of Teaching, chap. v. Houghton 
Mifflin Company, 1918. 

This chapter deals with the interpretation of scores yielded by the Courtis 
Standard Research Tests in Arithmetic, Series B, according to this pro- 
cedure. 

2 Zirbes, Laura. ‘“‘ Diagnostic Measurement as a Basis for Procedure”; in 
Elementary School Journal, vol. xvi, pp. 505-22 (March, 1918). 

This is an excellent illustration of the use of standardized objective tests 
in diagnosing pupils with respect to their achievements. 

Burgess, May Ayres. “Classroom Grouping for Silent Reading Drill” ; 
in Elementary School Journal, vol. xx, pp. 269-78 (December, 1921). 
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the capacity of the pupils to learn. All pupils belonging to 
a grade are judged with respect to the same norms. We 
know, however, that all pupils are not alike. Pupils belong- 
ing to a class frequently exhibit marked differences in their 
capacity to learn, although they are in some respects a se- 
lected group. Therefore it is not unlikely that two pupils 
belonging to the same class may make identical scores, but 
have quite different needs for remedial instruction because 
they differ in respect to their capacity to learn. For exam- 
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ple, suppose a “bright” pupil and an “average” pupil make 
the same score on a test on arithmetic problems. Suppose 
further that this score is just up to the norm for their grade. 
The “average” pupil has done as well as we have a right to 
expect him to do, but the “bright” pupil has not. He 
should make a much higher score because he possesses a 
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greater capacity to learn. He needs some sort of remedial 
instruction in order to bring his achievement up to the norm 
for his degree of intelligence. 

In order to take into account the general intelligence of 
pupils, mental-age norms have been proposed for use instead 
of grade norms.'!' The use of such norms makes it possible 
to compare the achievements of each pupil with the norms 
for his own mental age. Thus, he is judged with reference 
to his capacity to learn, rather than with reference to the 
norm for the grade in which he happens to be placed. It 
has been proposed that this comparison be made by dividing 
the pupil’s achievement by the norm for his mental age. 
The quotient obtained is called the achievement quotient.? 
If this is expressed as a per cent, a quotient of 100 means 
that the pupil’s achievement is just up to the standard for 
his mentalage. Ifthe quotient is less than 100, for example, 
80, it means that his achievement is only 80 per cent of his 
norm. Qn the other hand, if the achievement quotient is 
greater than 100, for example, 125, it means that his achieve- 
ment is 25 per cent above his own norm. 

Significance of achievement quotient. The significance 
of the achievement quotient in diagnosing individual pupils 
is illustrated in Fig. 5. In this figure the comprehension 
scores yielded by Monroe’s Standardized Silent Reading 
Test, Revised, are represented graphically on the horizon- 
tal scale. The vertical scale is for the purpose of represent- 
ing the achievement quotients. Each dot in the figure rep- 
resents a pupil, and its codrdinates are his comprehension 
score on this silent reading test and his achievement quo- 
tient. The horizontal position of the dot is determined by 
the pupil’s comprehension score on the test. The vertical 
position of the dot is determined by his achievement quo- 
tient. The grade norm for this class is approximately ten. 


1See page 179. - 2 I bid., 158. 
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It will be noted that a number of pupils made scores of ten. 
There is one score of sixteen, and one score of four. An 
achievement quotient of 100 indicates that the pupil’s 
achievement is up to the norm for his mental age. When the 
achievement quotient is less than 100 it indicates that his 
score is below the norm for his mental age. When it is 
greater than 100 the score is greater than the norm for his 
mental age. 

The highest achievement quotient in this class is earned by 
a pupil who made a score of eight on the test. He is two 
units below the grade norm, and would ordinarily be inter- 
preted as being below standard. The norm for his mental 
age, however, shows that he is not below standard but is 
decidedly above it. In fact his achievement in silent read- 
ing is approximately 80 per cent above the norm for his men- 
tal age. It is also interesting to note that the pupil who 
made the score of four, the lowest in the class, is shown to 
have an achievement above the norm for his mental age. It 
is impossible to know what a pupil’s instructional needs are 
until we know what his capacity to learn is. The achieve- 
ment quotient furnishes a convenient and effective way of 
recognizing the pupil’s mental age in interpreting his achieve- 
ment. 

General diagnosis vs. detailed diagnosis. Achievement 
within a school subject is complex and consists of many 
items, and in some cases these are relatively unrelated. A 
general diagnosis of pupils with respect to achievement 
involves simply locating pupils who have not achieved gen- 
eral objectives set for them. If we inquire into the particu- 
lar shortcomings of a pupil we have a detailed diagnosis. 
This may be extended somewhat indefinitely by the elabo- 
ration of our achievement tests. The two types of diagnosis 
are complimentary, and may be illustrated in the field of 
arithmetic. 
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The use of a general test upon the operations of arithmetic 
which includes examples from each of the four operations 
may be said to be diagnostic, in the sense of that it will 

-“spot” those pupils who failed to achieve in a general way 
in the field of the operations of arithmetic. The Courtis 
Standard Research Test in Arithmetic, Series B, is diagnostic 
in the sense that it measures achievements separately for 
each of the operations in the field of integers. It, of course, 
does not reveal a pupil’s particular limitations. It simply 
tells us the extent to which he is lacking in the general ability 
to do examples in each of the four operations. 

To ascertain the particular types of example which he is 
unable to do satisfactorily it is necessary to use a more 
elaborate series of tests, such as Monroe’s Diagnostic Tests 
in Arithmetic, or the Cleveland Survey Tests in Arithmetic. 
In diagnosing pupils with reference to achievement it is prof- 
itable for teachers to use a test, or battery of test, which 
yield detailed information concerning the achievements of 
pupils. The time and money at a teacher’s disposal, of 
course, places certain limitations upon the elaborateness of 
the testing program. In some cases it is advisable to give 
a general test first and then supplement it, in the case of 
those pupils who exhibit general shortcomings, with a more 
detailed test. 

Analytical diagnosis. A diagnostic test can do no more 
than yield detailed information concerning a pupil’s achieve- 
ments. In most cases a study of his scores does not tell us 
why he failed to achieve or what particular errors he has 
made. In planning effective remedial instruction it is fre- 
quently helpful to secure this information by means of ana~ 
lyzing the pupil’s performance. This is going beyond meas- 
urement in the sense implied by the ordinary use of educa- 
tional tests. An analytical diagnosis, however, is frequently 
profitable, particularly in the case of those pupils who are 
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found to be lacking in achievement. It may be based upon 
the performances recorded in response to a standardized edu- 
cational test, or it may be based upon performances secured 
in other ways.! 


5. Useful remedial measures 

Planning remedial instruction. A detailed consideration 
of remedial instruction is beyond the scope of this volume. 
When a pupil fails to achieve up to the norm for his mental 
age he does so for some reason. Barring physical defects 
and outside influences, the cause is in the instruction he has 
received. He may need to be given a stronger motive. He 
may need more instruction, or another kind of instruction. 
It may be that he has not mastered properly some of the 
prerequisites for this particular subject. Whatever the 
causes are, the teacher’s problem is to remedy the undesirable 
condition, Measurement by means of standardized ob- 
jective tests can only reveal the pupils who are lacking in 
achievement. They do not reveal the causes. To ascer- 
tain the exact cause, and to formulate remedial instruction 
which will bring the pupil up to the norm for his mental age, 
will tax a teacher’s resourcefulness and his acquaintance 
with the principles of instruction. No detailed directions 
can be given. The teacher must rely upon his acquaintance ° 
with the process of learning and upon his experience. The 
tests only reveal the pupils who have need for remedial in- 
struction. 

Diagnosis of pupils with reference to their study proce- 
dure. The diagnosis of pupils with reference to study pro- 
cedure is not fundamentally different from diagnosis with 


1 Monroe, W.S. Measuring the Results of Teaching, pp. 188-52. Hough- 
ton Mifflin Company, 1918. 

This gives a summary of several reports of analytical diagnosis in the 
operations of arithmetic. 
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reference to achievement. In fact, ability to study effec- 
tively is one form of achievement. As we indicated in Chap- 
ter III, in the upper grades and in the high school the diag- 
nosis of pupils with respect to their study procedures to a 
large extent takes the place of diagnosis with respect to 
achievement. In the diagnosis of study procedure one is 
concerned with locating the pupils who have not acquired 
effective methods of study. It is not a question of deter- 
mining who has achieved and who has not, but who has 
acquired a procedure which makes achievement possible and 
who has not acquired it. For exarple, one might have ac- 
quired a good procedure for studying mathematics but at 
the same time have learned very little mathematics, because 
he had studied mathematics very little or not at all. In 
addition to being a prerequisite for achievement, effective 
methods of study may be considered as objectives. They 
are among the most important outcomes to be realized by 
the school. 

Relatively few tests are available for securing the infor- 
mation which is necessary in diagnosing pupils with respect 
to their study habits. Since reading is the basic activity ina 
number of fields of study, a silent reading test will throw 
some light upon pupils’ study procedures. 

Setting immediate educational objectives. General state- 
ments of educational objectives are not sufficient to provide 
pupils with definite goals for their daily work. There is 
need for immediate and detailed educational objectives that 
can be understood by the pupils. In so far as the norms 
derived for our educational tests represent degrees of ability 
that should be attained, they may be used as objectives. 
Since they are stated in definite terms, and in the case of 
many of our tests are given in detail, they naturally become 


a very satisfactory statement of immediate educational ob- 
jectives. © 
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For example, if we assume that the norms for the Courtis 
Standard Research Tests, Series B, represent the degrees of 
attainment which we should expect of pupils in the opera- 
tions of arithmetic with integers, these norms may then be 
used as immediate educational objectives in this field. They 
can be easily understood by the pupils. They know that their 
goal is to do so many exercises of a given type in a given time, 
with a certain per cent of their answers correct. Similar 
norms can be stated for handwriting, reading, and in fact 
for all subject-matter fields in which we have objective tests 
whose norms may be considered to represent satisfactory 
objectives. 

Motivating school work through the use of standardized 
objective tests. The setting of definite immediate objectives 
is an effective method of creating a motive. When a group 
of pupils is given a rate of 200 words per minute as an im- 
mediate objective in silent reading they are able to under- 
stand what the objective means. They have something very 
definite to work for. A similar condition exists when they 
are given an objective in handwriting, such as a rate of 80 
letters per minute with a quality of 60 on the Ayres Hand- 
writing Scale. The mere fact that they are given objectives 
which they can understand and which are definite accounts 
for the motivation in part. This motivation is, however, in- 
creased by a knowledge that instruments are available which 
can be used to measure their achievement in any stage of 
their learning. In the case of some subjects, such as hand- 
writing, these instruments may be placed in the hands of the 
pupils, and they can ascertain from time to time exactly 
where they stand with reference to the objectives that have 
been set. 

When definite immediate objectives are set by a teacher, 
and an effort is made to have the pupils attain these objec- 
tives, unusual gains in achievement are frequently made. 
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The following illustration is typical. Just after the mid- 
year promotion Monroe’s Silent Reading Tests were given 
to the pupils in grades 3-8 inclusive in one building in a 
large city. Both Forms 1 and 3 were given at this time. 


COMPREHENSION 


JUNE MEDIAN SCORES see 
Gade (aS. 
FEB. fer SCORES 


HIB IIA IVB IVA VB VA VIB VIA VIB VIIA VIIIB VIIA 
GRADES 


Fia. 6. Garns IN CoMPREHENSION OF SILENT READING DUE TO 
DeFIniTE InstRUCTION FOR THIS PURPOSE 


The median scores showed that in most grades the average 
attainments of the pupils in silent reading were below the 
norms for these tests. The teachers of this school adopted 
the norms of the tests as educational objectives. Without 
devoting any more time to the teaching of reading, a definite 
effort nevertheless was made to train the pupils in rate and 
comprehension of silent reading. Just before the close of the 
school year, in June, Forms 1, 2, and 3 of Monroe’s Stand- 
ardized Silent Reading Tests were given to these pupils. 
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RATE-WORDS PER MINUTE 


JUNE STANDARDS |— —— 


| 


JUNE a SCORES 
TANDARDS 


FEBRUARY 


re) 
11B HA IVB IVA VB VA VIB VIA VIIB VIIA VIIIB VIIIA 
GRADES 


Fic. 7. Gans In Rate or Sitent Reapine Due To DerFinitE 
INSTRUCTION FOR THIS PURPOSE 


In Figures 6 and 7 the average median scores ! and the stand- 
ard scores both for February and June are represented 
1 Although the three forms of Monroe’s Standardized Silent Reading 
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graphically. The average median scores are represented by 
solid lines in both figures; the lower one being for February 
and the upper one for June. The broken lines represent 
norms for February and June. 

These two figures show that in each grade relatively large 
gains were made in both rate and comprehension. These 
gains in general are considerably greater than the normal 
gain for the period between February and June. In some of 
the grades it is four or five times as great. The gains shown 
were doubtless affected somewhat by the increased acquaint- 
ance of the pupils with the tests and with testing conditions. 
There is no doubt, though, that the adoption of the norms 
for these tests as educational objectives greatly motivated 
and directed the school work, so that unusual progress was 
made by the pupils in the field of silent reading. 

Reporting the achievements of pupils to parents. Since 
scores are more accurate measures of achievement than the 


Tests were designed to be equivalent, investigation has shown that they are 
not equivalent. Pupils make somewhat higher scores upon Forms 2 and 3 
than they do upon Form 1. It is estimated that multiplication by the fol- 
lowing factors will reduce scores made on these forms to the equivalence 
of Form 1 scores: 


Form 2, Comprehension Test I .97 Test II .94 
Rate Soi 4/88 YT iLeess 
Form 8, Comprehension Test I 1.00 Test II .88 
Rate Seok 280 “ TI .93in grades 6and 7. 


.95 in grade 8. 
The median scores for Forms 2 and 3 were multiplied by these factors be- 
fore they were averaged. 

In reporting these scores graphically it was necessary to recognize the 
fact that grades III-B to V-A were given Test I, and Grades VI-B to VIII- 
A were given Test II. Although it was intended that these two tests should 
have the same size units anda common zero point, experience has shown 
that they do not have. It is estimated that in order to make the com- 
prehension scores obtained from Test II comparable with those ob- 
tained from Test I, three units must be added to the Test II scores. In 
the case of rate, it is necessary to add 10 units. This has been done in 
drawing the graphs so that the lines would show uniform development. 
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usual school marks, it has been proposed that the scores 
yielded by the standardized educational tests be entered 
upon the reports sent to parents. As we have previously 
pointed out, measures of achievement can be interpreted 
only by comparison with appropriate norms. Thus, if the 
scores yielded by standardized educational tests are entered 
upon the reports sent to parents it will be necessary to enter 
also appropriate norms. This can be avoided to some ex- 
tent if the derived measures, described in Chapter VII, are 
used instead of the point scores. In any case it will be nec- 
essary to supply patrons of the school with the information 
needed for an intelligent interpretation of the scores. 

It should be remembered that our standardized objective 
tests do not yield perfectly accurate measures of achieve- 
ment. Although they do not appear to involve as large 
errors of measurement as ordinary school marks, it is im- 
portant that both children and parents should not be given 
the impression that the scores are perfectly accurate meas- 
ures of achievement. If the scores are presented as being 
ideal measures of achievement both pupils and parents 
will be inclined to make misleading interpretations. It has 
been noted in several connections that measures of achieve- 
ment cannot be completely interpreted without a knowledge 
of the pupil’s general intelligence. There is some doubt 
whether at the present time it is desirable to report measures 
of general intelligence to children and their parents. For 
this reason the use of the results yielded by standardized 
objective tests should be used with caution in reporting the 
achievements of pupils to parents. 


II. Uses or STANDARDIZED TESTS BY THE SUPERVISOR 

Six major supervisory uses for tests. The major activ- 
ities in which the supervisor may use standardized educa- 
tional tests are the following: 
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1. Promotion and classification of pupils. 

2. Educational and vocational guidance. . 

3. Supervision of the instruction including the evaluation 

of school efficiency. 

4. Rating of teachers. (This is partly included in the 

above activity.) 

5. School publicity. 

6. Scientific experimentation. 

In the first two of these activities the supervisor has to 
deal with individual pupils. Therefore, he will have need 
for standardized educational tests that will yield reliable 
individual measures. In the other four activities mentioned 
the supervisor is dealing for the most part with groups of 
pupils. When he wishes to obtain only a general survey or 
average measure of the school system, or some division of 
it, he may use a test which does not yield accurate measures 
of individual pupils. Since the probable variable error of 
measurement is inversionally proportional to the square 
root of the number of pupils in the group, the measure of a 
group of pupils will always be more accurate than the meas- 
ure of a single pupil. Certain very brief tests have been 
designed for the use of supervisors. This is true of the Cour- 
tis Supervisory Tests in Arithmetic, Monroe’s General 
Survey Scale in Arithmetic, the Indiana Scales of Attain- 
ment, No. 2 and No. 1, and the Pintner Survey Test. 


1. Promotion and classification of pupils 

Grade placement of pupils entering school. When pupils 
enter school for the first time they are usually placed in the 
first grade, and all are required to do the same work. Re- 
cently it has been suggested that when a school can be or- 
ganized so as to permit of it, the pupils entering should be 
grouped according to their capacity to learn. This would 
mean that some would be placed in I-A, others in II-B, and 
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possibly some in I-A. In one school system, that has come 
to the attention of the writer, pupils below a certain chron- 
ological age are not admitted to school until they have 
attained a certain mental age. In placing pupils who are 
entering school, general intelligence tests are almost indis- 
pensable. Since the pupils are just entering the school the 
supervisor has very little information concerning them. A 
few minutes devoted to the application of a general intelli- 
gence test will enable the supervisor or the teacher to place 
such children with surprising accuracy. 

Placement of transfer pupils. Pupils frequently transfer 
from one school to another and when the courses of study 
followed in the two schools are not identical a problem fre- 
quently arises in placing such pupils. Standardized educa- 
tional tests can be used to measure their achievements. A 
general intelligence test will yield a measure of their capa- 
city to learn. Comparison of the measures with the grade 
norms will indicate where a pupil should be placed. Of 
course any additional data, such as his school record, chron- 
ological age, health, etc., should be taken into considera- 
tion. 

Promotion of pupils. In practically all school systems 
pupils are promoted from one grade to another at specified 
intervals. The rules governing the promotion of pupils is a 
matter of school policy. What the passing mark should be; 
whether the promotion is to be made upon the basis of capa- 
city to learn or upon the basis of past achievements; whether 
promotion will be on trial or shall be based solely upon pre- 
vious work; what shall be done with doubtful cases; these 
are questions to be settled by the supervisor and the in- 
structional staff of the school. No standardized educational 
tests can answer any of these questions of school policy, but 
after the general policy has been defined standardized edu- 
cational tests are valuable instruments in dealing with some 
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of the problems that arise in connection with the promotion 
of pupils. They have frequently revealed bright pupils 
who had not been suspected of this capacity by their teach- 
ers. They yield valuable supplementary information in 
doubtful cases. 


TABLE X. PER CENT OF PUPILS FAILING IN ONE LARGE 
BUILDING OF A CITY SCHOOL SYSTEM 


. Per cent of failures 
esd 6B GA 
Reading in cctoccc ers ¢ ocicis ects seine es clginatcie salar aia'ele'ela 20 ve 
ATith Metis’ ais siiccislee'cicca ic iee'e sieinietalerele siclerele cleine/a 35 36 
Spellinge. i taciscsrraes sleiclale ap ajleieleteieiersiersiaiel ae aieialeisieter 10 13 
IPGnimansulpetes ce acrelactsianrceie tiare celcicleleiererciticietecsre eine 42 13 
ERistory eerie atcic! eve Sleislcte cre clereletclelercierelatersielalelereleze 31 @4 
GeO gta POY a ietia cals !s:aic\e/siaicie 4 eisierejaie eee slaretavelciateietela (aia 28 27 
Oral Compositions; co's. veces ee Cras ee cle Caceserctae 85 22 


Written Composition 


It is important that there shall be uniformity with respect 
to the promotion of pupils within a school system. How- 
ever, it not infrequently happens in a large city school sys- 
tem that some teacher or group of teachers may not conform 
to the general policy. Standardized objective tests furnish a 
means for investigating the promotion record of a teacher 
or of a school. For example, in one school building in a 
large city school system the per cents of failures were very 
high in VI-B and VI-A. They are given in Table X._ Be- 
cause such a large proportion of the pupils in these two half 
grades received a mark of failure the supervisor naturally 
wondered whether the pupils in this school were so inferior 
in capacity to learn that an unusually large per cent of them 
should fail. 

In order to answer this question the [linois Examination, 
including tests on general intelligence, the operations of 
arithmetic, and silent reading, was given to the pupils. The 
scores obtained showed clearly that the pupils in these two 
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half grades were on the average somewhat superior to pupils 
in the corresponding grades in other schools. They were 
shown to be superior not only in general intelligence but 
also in the operations of arithmetic and silent reading. The 
median achievement quotients were distinctly above 100, 
and also above the medians in other schools in this city. In 
fact these standardized educational tests showed that the 
pupils in these two half grades were not inferior in either 
capacity to learn or in their achievements. Consequently, 
the supervisor was able to assert that the high per cent of 
failures was not in agreement with the general promotion 
policy of that school system. 

Classification of pupils within a grade. Pupils belonging 
to the same school grade are known to differ widely with 
respect to their capacities to learn. Some are dull, others 
are bright. Even when pupils are placed in the grades that 
correspond to their mental ages, there will still be differences 
in I.Q.’s. Those having high I.Q.’s will be able to learn 
more rapidly than those who have low I.Q.’s._ In school 
buildings where there are two or more teachers giving in- 
struction within the same grade, and to a limited extent in 
smaller buildings, it is possible to classify the pupils belong- 
ing to a school grade for purposes of instruction. 

It has been proposed that the efficiency of a school will 
be increased if the pupils are classified so that those possess- 
ing approximately the same degree of brightness will be in- 
structed together. ‘The instructional groups will then be 
more homogeneous. Approximately the same work may 
properly be expected from all members of the class. There 
will be no bright pupils to be bored because of the slow and 
monotonous pace that is set for the less capable pupils, and 
there will be no dull pupils who are unable to keep up be- 
cause they do not have the capacity to learn as rapidly as 
the average of the class. This proposal is supported by a 
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great many supervisors and teachers who have put it into 
practice. 

General intelligence tests, supplemented by achievement 
tests, afford a basis for making such a classification of pupils. 
The classification should not be made on the basis of gen- 
eral intelligence alone, because it is necessary to consider 
the attitude of a pupil toward the work of the school as well 
as a number of other factors which affect his achievement. 
A pupil’s mental age and intelligence quotient are, how- 
ever, the two most significant items with reference to the 
classification of pupils for instructional purposes. 

Selection of exceptional children. Some children differ 
so greatly from the average child of the same chronological 
age that special instruction is required. Exceptionally dull 
or backward children need to be placed in special classes. 
Pupils who are exceptionally bright also demand special 
instruction. In fact the provision of appropriate training for 
such children is much more profitable to society than the 
training of dull or backward children. For the selection of 
both types of exceptional children general intelligence tests 
are indispensable. Tests for measuring the achievement of 
pupils, particularly in the case of exceptionally bright chil- 
dren, are also very valuable. 


2, Educational and vocational guidance 


Use of tests in this work. It is now the consensus of 
opinion that the school should place the pupil and guide 
him in the selection of his work so that he will secure the 
maximum returns for the time he spends in school. The 
promotion and classification of pupils, which has been dis- 
cussed above, is one form of educational guidance. How- 
ever, educational guidance is coming to mean more than 
this. Terman and others have shown that pupils below a 
certain mental level have very small chances for success in 
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certain school subjects. In the elementary school there is 
little opportunity for the selection of subjects to be studied 
below the sixth grade. Practically all pupils must devote 
their time to the study of minimum essentials. Here educa- 
tional guidance is confined almost exclusively to the promo- 
tion and classification of pupils. In the junior high school, 
and to a greater extent in the senior high school, some elec- 
tion of work is possible. 

It is not expected that all pupils will study the same sub- 
jects. In making their selections pupils should be advised 
with reference to the courses which they may undertake 
with a reasonable chance of success. They should be guided 
in part by their interests, but it is unwise for pupils to under- 
take subjects in which the chances for success are very lim- 
ited. This guidance should be based upon definite informa- 
tion concerning a pupil’s capacity to learn, and upon his 
achievement in the past. Standardized educational tests 
are valuable instruments for this purpose. The scores ob- 
tained should be interpreted with reference to success norms, 


8. The supervision of instruction 

Objective measuring substituted for subjective estimat- 
ing. A supervisor must keep in touch with his school sys- 
tem. From time to time it is imperative that he make a sur- 
vey of it. He needs to compare the efficiency of his system 
with that of other similar systems. He also needs to be in- 
formed of the relative efficiency of the units of his own sys- 
tem. Before standardized educational tests were available, 
such surveys were made by a personal visitation. Obvi- 
ously they were subjective. Standardized objective tests 
afford a means for making an objective survey. 

The supervisor diagnoses the school system in much the 
same way that the teacher diagnoses his class. The gen- 
eral procedure is not materially different. It has been 
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shown that the most effective method of supervising instruc- 
tion is to ascertain the divisions of the school system which 
are below the desired standard of efficiency, and to prescribe 
definite remedies for undesirable conditions that are found 
to exist.1. In this way the efforts of a supervisor will be 
directed toward the meeting of specific needs. In the ab- 
sence of a diagnosis of his school system the supervisor will 
frequently direct his energies in ways that are not profitable. 
It may be that a reorganization of the system is needed. 
The course of study may be lacking. The teachers may need 
some special training. Whatever the needs are the super- 
visor will be in a better position to meet them when he is 
acquainted with the conditions that exist. 

Evaluating the efficiency of a school system.? Standard- 
ized educational tests are sometimes mentioned as instru- 
ments which can be used to measure the “efficiency” of a 
school system, or some division of it. Such statements 
are misleading. A standardized educational test may be 
used in arriving at a measure of “efficiency,” but one test, 
or even a group of tests, cannot yield direct measures of it. 
**Efficiency,”’ as used in education, has a meaning which is 
not essentially different from that which the term has when 
applied to an industrial project, or even to a single machine. 
In these fields “efficiency” is a ratio whose maximum value 
is 1.00. This ratio is a fraction, whose numerator is the 
output and whose denominator is the input or investment. 
In education, the output consists of the changes produced 
in the pupils; i.e., the controls of conduct that the school 
engenders. The output for the year is the total of all the 

1 Courtis, S. A. “ Measuring the Efficiency of Supervision in Geography ”; 
in School and Society, vol. x, pp. 61-70 (July 19, 1919). 

2 The reader should distinguish between the “efficiency of a school 
system” discussed here and the “efficiency of instruction ”’ considered on 


page 172. The term “efficiency” as used here has a more complete 
meaning. 
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changes ! that have been produced in the pupils during the 
year, due to the influence of the school. The educational 
investment includes many factors, such as buildings, equip- 
ment, textbooks, teachers, supervision, and general ad- 
ministration. It also includes the time during which these 
investmental factors operate. The ratio is also affected by 
the quality of the pupil material, organization of school sys- 
tem, methods of instruction, and so forth. 

A standardized educational test measures only the achieve- 
ments of pupils in the field of a school subject. If we as- 
sume that the achievements measured by a given test are 
representative of the total field of a school subject, we may 
consider that we have an average measure of all of the pupils’ 
achievements in this field. The general intelligence of the 
pupils must also be measured. Even when we have meas- 
ured this, we still do not have a measure of the efficiency of 
the school system, because no account has been taken of the 
investment in buildings, equipment, textbooks, teachers, 
supervision, and general administration, and of the time dur- 
ing which these factors operated. Hence, a single standard- 
ized educational test, or even a group of standardized edu- 
cational tests, cannot yield, directly, measures of the effi- 
ciency of a school system. 

Efficiency measurement calls for intelligence measure- 
ment also. The necessity for securing measures of general 
intelligence of pupils is emphasized in the following illus- 
tration. Monroe’s Standardized Silent Reading Test, and 
Monroe’s General Survey Scale in Arithmetic, were given 
throughout two school systems. The two cities concerned 
are approximately the same size, and do not differ markedly 
with respect to the character of their population. The 
median grade achievements of the two cities are shown in 
Fig. 8. It is clearly evident in this figure that City B is dis- 

~ 1A more exact statement would be the social worth of these changes. 
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tinctly superior to City A in both arithmetic and reading, 
when compared grade for grade. The superiority is more 
pronounced in the case of reading. If this were the only in- 
formation at hand for the two cities we would have to con- 
clude that the school system of City B was distinctly supe- 
rior to that of City A, and that City A was distinctly below 
the city average. 


CITY AVER.. 
CITY A. 
CITY B. 


Fic. 8. Acnievement AcEs ror ARITHMETIC AND FoR READING 
(AvERAGE) For City A anp City B 


Fortunately, the Illinois General Intelligence Scale was 
administered to all pupils at the same time that the reading 
and arithmetic tests were given. The median mental age 
and the median I.Q. are given for these two cities in Fig. 9. 
This figure shows that the school system in City B has 
been organized so that in each grade the median mental age 
for the pupils is about one year in advance of the median 
mental age of the pupils of the corresponding grade in City 
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A. The median I.Q.’s are also distinctly higher. In fact in 
City B the median mental age of the pupils is shown to be 
distinctly above the city average. These facts mean that, 
grade for grade, City B has pupil material that is distinctly 
superior to that in City A. This is probably not due to any 
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Fic. 9. Mepran Mentat Ace AnD Meptan I.Q. ror Eacu GrapE 
For City A anp City B 


inherent differences in the quality of the pupils in the two 
cities, but rather to the differences in the organization of the 
two school systems. In City B the school system has been 
organized so that the pupils, particularly in the upper grades, 
are highly selected. The promotion rate has been relatively 
low, and pupils who did not get along well in school have 
dropped out after passing the age limit of compulsory attend- 
ance. The opposite policy has predominated in City A. 
This system has been liberal in regard to promotion, and 
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there has been an effort to keep pupils in school. This has 
resulted in less highly selected groups of pupils, particu- 
larly in the upper grades. 

In Fig. 10 the median achievement quotients are shown 
for these cities. In the case of arithmetic, the median 
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Fig. 10. ACHIEVEMENT QuoTIEnTs ror Eacn Grape For City A 
AND City B 


achievement quotients for City B are distinctly below those 
for City A, except in the eighth grade. They are also below 
city average. In the case of silent reading, City B sur- 
passes City A only in the seventh and eighth grades. There- 
fore, we conclude that, with the exception of the eighth 
grade and in part the seventh, the school system of City B 
is less efficient in teaching the operations of arithmetic and 
silent reading than the school system of City A. This con- 
clusion is the opposite of the one suggested by the absolute 
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measures of achievement. By having at hand a measure 
of the mental ages of the pupils, it has thus been possible 
to avoid an erroneous interpretation of the measures of 
achievements for these two cities. 

The need for measures of the general intelligence of pupils 
in interpreting measures of their achievements exists also in 
studying a division of a school system. In fact the need is 
even greater, because the quality of the pupil material in 
different sections of the city is likely to exhibit greater differ- 
ences than we find between cities. 

When interpreting the measures yielded by standardized 
educational tests with reference to the efficiency of the 
school, it is necessary to bear in mind the various factors 
which affect the achievements of pupils. General intelli- 
gence, as we have just shown, is a very potent factor. Hered- 
ity, maturity, and general training affect achievements of 
pupils as well as the specific training which the school affords. 
Nationality, race, environmental conditions, and perhaps 
some other factors must be considered in certain cases, 
Heredity, maturity, general training, nationality, and race 
will generally be sufficiently accounted for in the measures of 
general intelligence, but under certain conditions they may 
require special consideration. Courtis ' has shown that ma- 
turity and general training appear to be the most potent 
factors affecting the development of certain abilities. If 
this is true, the measurement of such abilities can give no 
indications of the efficiency of the training which the school 
provides. 

The use of comparative data. In interpreting the median 
scores for a city comparisons are frequently made with sim- 
ilar scores from other cities. In our discussion of norms, 
in Chapter VIII, we urged certain cautions in their use. In 


1Courtis, S. A. Measurement of Classroom Productions, pp. 357 ff, 
Report of Gary School Survey, General Education Board, 1919. 
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the use of other comparative data for judging the relative 
efficiency of a school system it is even more important that 
these cautions be kept in mind. We have just pointed out 
the necessity of taking into account the general intelligence 
of the corresponding groups of pupils in the cities concerned. 
It is also necessary to take into consideration such factors as 
recency of instruction on the content of the test used, the 
familiarity of the pupils with the testing procedure, their 
acquaintance with the particular test used, the teaching 
time allotted to the subject in which the test is given, and 
the distribution of the time within this subject. 

We have noted in another connection that second-trial 
scores, when a test is repeated after a time interval of not 
more than a few days, are likely to be as much as 10 per cent 
greater than first-trial scores. When a test is repeated after 
a longer interval no definite statement can be made, because 
the difference between the scores obtained from the two 
trials will depend upon the instruction which the pupils 
have received during this interval. If some of the compara- 
tive data represent first-trial scores, while others are from 
second- or third-trial scores, the data cannot be really con- 
sidered comparable. Also, corresponding scores from two 
cities cannot be considered to be comparable unless the corre- 
sponding groups of pupils are equivalent in general intelli- 
gence, and have received equivalent amounts of instruction. 
Hence, it is dangerous to compare the median scores ob-. 
tained for one school with the median scores reported for 
other schools when nothing is known concerning the factors 
which materially affect the achievements of pupils. 

The measurement of progress. Standardized educa- 
tional tests have been recommended as instruments which 
a teacher or supervisor may use to secure a measure of the 
progress for the year or semester. It has been recommended 
that a suitable test be given at the beginning of the year or. 
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beginning of the semester, and that this test be repeated at 
the end of the year, using a duplicate form if possible. The 
differences between the two sets of median or average scores 
have been considered to represent the progress made during 
the interval of time that had elapsed from the first to the 
second testing. This procedure has been suggested as a 
method whereby a supervisor might not only keep in touch 
with the efficiency of his system, but also secure significant 
measures of the effectiveness of his teachers. A teacher 
might use it to secure a measure of his own efficiency. 
Important modifying factors. The score which the pupil 
receives on a test depends upon a large number of factors 
other than his ability. The differences between two sets 
of median scores are accurate indices of increase in achieve- 
ment only if all factors other than ability are the same at 
both testing periods. If the pupils are unacquainted with the 
test, particularly if they are unacquainted with the testing 
procedure, this condition cannot exist at the second testing. 
They will have become acquainted with the test through the 
giving of it at the first testing period. We urge teachers to 
use standardized educational tests as a means for diagnos- 
ing their pupils. In this connection, it is very properly 
pointed out that unless something is done to remedy the 
conditions revealed the giving of the tests cannot have much 
value. This means that the teachers should take steps to 
apply remedial instruction where the need for it has been 
shown. In doing this, the pupils will be made acquainted 
with the types of exercises which the tests include, unless they 
did them satisfactorily the first time. Naturally, this will 
tend to make materially higher scores on the second trial. 
None of our tests may be considered to be all comprehensive, 
even though they are representative of the field of a school 
subject. The teacher who attempts merely to correct the 
defects revealed at the first testing will have a group of pupils 
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who will show marked increases. However, these increases 
are likely not to be representative of the real progress of the 
pupils in the field of the subject. Another teacher who did 
not emphasize the remedial instruction to the exclusion of 
instruction on other topics would show smaller differences 
between the two sets of scores, but actually might have 
achieved greater progress. 

The recency of instruction also materially affects the 
scores. If the pupils had received no recent imstruction 
upon the exact content of a test at the time of the first trial, 
but had received instruction on it just preceding a second 
trial, large differences would be shown, but they would not 
be indicative of real progress. In order to offset the effect 
of recent instruction, it has been proposed that a measure of 
the progress for a year be secured by comparing the scores 
made by pupils just after the beginning of one school year, 
with those made just after the beginning of the next school 
year. Any increase shown would be a permanent increase, 
because it would be the residue remaining after the summer 
vacation. The measurement of progress in this way will 
probably be more accurate than that secured by repeating 
the test at the close of the school year. 

Although the comparisons of the scores made at the be- 
ginning of the school year with those made at the end of a 
school year, or, better still, with those made at the begin- 
ning of the following school year, yield some measure of 
progress, it is necessary to exercise caution in interpreting 
these comparisons. It is very easy to do an injustice to a 
teacher or to an entire school. So many factors affect a 
pupil’s score that these differences may be due entirely to 
causes other than real growth. 


4. The rating of teachers 
Standardized obiective tests used in rating teachers, 
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This is essentially a special case of evaluating the efficiency 
of a unit of a school system, but because of its importance 
it deserves special treatment. In some school systems 
increases in salary and promotion depend upon the ratings 
given to teachers. Since teachers are employed for the 
purpose of producing certain changes in pupils placed under 
their instruction, it has been suggested that standardized 
objective tests may be used to measure these changes, and 
hence to provide an objective measure of the teacher’s 
efficiency. | 

According to this proposal, by giving several tests at the 
beginning of the year a supervisor will be provided with a 
statement of the present status of the pupils. When the 
tests are repeated at the close of the school year the differ- 
ences between the two sets of scores will be an index of teach- 
ing efficiency. If an extensive battery of standardized edu- 
cational tests is used a supervisor will be provided with a 
measure of the extent to which the teacher has fulfilled his 
function. 

Limitations of the method. This proposal makes a strong 
logical appeal. Teachers should be judged by the results 
which they produce. There are, however, some reasons why 
standardized educational tests should not be given a large 
place in the rating of teachers. In the first place, we do not 
have available standardized educational tests for measuring 
all of the changes which teachers are expected to produce in 
pupils. We have been most successful in measuring skills, 
and information in the fields of arithmetic, reading, writing, 
spelling, ete. These changes in pupils are fundamental, and 
the teacher who is not effective in producing these changes 
cannot be counted successful. But there are many other 
important outcomes. Many of these we are as yet unable 
to measure. We would be unjust to the teacher, and incon- 
sistent with our educational objectives, if we did not recogs 
nize this fact in judging a teacher’s efficiency. 
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The second reason is the errors which may be introduced 
into our measurements. We have already discussed these 
under the heading of “measurement of progress.”” Hence, 

‘it is unnecessary to indicate the nature and magnitude of 
these errors here. It is sufficient to say that the errors may 
be large enough to distort seriously a rating given to a teacher 
on the basis of the scores yielded by standardized educa- 
tional tests. Thus, although standardized educational tests 
may sometimes be helpful in the rating of teachers, the wise 
supervisor will use them with caution and with due regard 
to their limitations. 


5. School publicity 


Use, and limitations. The measures of achievements 
yielded by standardized educational tests of pupils lend 
themselves to publicity. They are definite numerical facts. 
When they are set beside the corresponding norms, or sim- 
ilar data from other school systems, they will appeal to 
many patrons of the school. Generally the patrons will be 
interested, especially, if their school is shown to stand high, 
or the results of the tests tend to corroborate their opinions. 

We have already said enough in other connections con- 
cerning the limitations of standardized educational tests to 
make it clear that a supervisor should be cautious in pre- 
senting the scores obtained. In addressing a lay audience 
it is probably unwise to emphasize the errors involved in our 
present measurements, but the tests should not be presented 
as instruments yielding highly accurate measures. To the 
layman accuracy has a meaning derived from the measure- 
ment of physical objects. A test may be highly accurate 
among tests, but in comparison with instruments for measur- 
ing physical objects all of our tests are very inaccurate. 

When a supervisor presents the results of a testing pro- 
gram to his constituency he must remember that his audi- 
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ence is not critical. They are relatively unacquainted with 
the tests used and their limitations. To them he is vouch- 
ing for the interpretations as well as the validity of the 
facts. The way in which the facts and the comparative 
data which accompanies them are presented influences to a 
large extent the interpretation. The supervisor should, 
therefore, be careful to present the facts so that they will be 
properly interpreted. 


6. Scientific experimentation 

_ Every school is a potential laboratory. Both the teacher 
and supervisor have an opportunity to determine the value 
of the methods and devices of teaching, and other items of 
school procedure. The carrying on of an investigation will 
tend to arouse the interest of teachers in school, and will 
also tend to stimulate them to think about their work. Even 
if little is contributed to the accumulation of human knowl- 
edge the work will frequently be very profitable. However, 
a supervisor should bear in mind that the school exists for 
the education of children and not to provide a laboratory for 
scientific experimentation. Occasionally it may be worth 
while to create artificial conditions for the purpose of study- 
ing certain items of school procedure, but frequently it will 
be unwise to control conditions to the extent that is required 
for careful scientific experimentation. 

Every experiment should have a well-defined purpose. It 
should be directed toward the solution of some educational 
problem and not merely for the purpose of satisfying curi- 
osity. Sometimes one or more educational tests are given to 
a group of pupils without any particular purpose in mind 
and later an attempt is made to utilize the data collected. 


1 Monroe, Walter S. Measuring the Results of Teaching, chap. um. 
Houghton Mifflin Company, 1918. 
This chapter suggests some effective methods of graphical representation. 
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Such a procedure will seldom lead to results that are signifi- 
cant. In so far as it is possible the experiment should be 
planned in detail before it is begun. It is necessary to guard 
against not only the errors which are likely to occur in the 
scores yielded by the tests used, but also other experimental 
conditions which may seriously limit the validity of the con- 
clusions. 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1. What is diagnosis? Describe the different kinds. 

%. How would you go about choosing a test? What would you need 
to know in order to make an intelligent selection? 

8. Plan a testing program for the elementary schools in a city of 2000. 
In acity of 5000. Ina city of 50,000. Go into detail. 

4. Plan a testing program for a high school of 200 pupils. 

5. Why is the efficiency of a school not measured by a standardized 
educational test? 

6. Plan an experiment in which educational tests would be used, and in= 
dicate the precautions you would take to secure accurate results. 

7%. Distinguish between “errors of measurement” and “‘errors of in- 
terpretation.” 

8. To what extent might the results of educational testing be used in a 
rating scheme for teachers? 

9. Can the giving of an occasional test without any particular purpose in 
mind be justified? If so, on what grounds? 

10. Should teachers and supervisors be encouraged to use educational 

tests before they have become acquainted with their limitations? © 


CHAPTER XI 
THE IMPROVEMENT OF EXAMINATIONS 


Examinations not completely replaced by standardized edu- 
cational tests. In Chapter II examinations set by a teacher 
or other school official were shown to yield highly inaccurate 
measures of school achievement, and to be subject to certain 
other limitations. Beginning with Chapter IV, we have dis- 
cussed in detail the construction of standardized educational 
tests in which these defects have been eliminated or re- 
ducedto a minimum. ‘These instruments for measuring the 
achievements of school children have become very widely 
used, particularly in the elementary school, but examina- 
tions set by the teacher remain the most frequently used 
means of measuring the achievement of pupils. Although 
properly constructed standardized educational tests are 
superior in certain respects to examinations set by the 
teacher, they will probably never entirely replace them as a 
means of measuring the results of teaching. 

At the present time we have satisfactory standardized edu- 
cational tests for only a few subject-matter fields. In the 
high school, with two or three exceptions, we have no tests 
which may be thought of as representative of the entire 
field of a school subject. Even in the elementary school, 
except for the tool subjects, we have relatively few satisfac- 
tory standardized educational tests which are sufficiently 
broad in their scope to be used as representative of a school 
subject. Furthermore, the teacher frequently has need for 
a measuring instrument adapted to a particular course of 
study, or to the emphasis which has been given to the sub- 
ject in the teaching of a particular class. For certain pur- 
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poses a teacher is justified in departing from the official 
course of study. When this is done there is need for an in- 
strument to measure the achievements of his pupils. A 
standardized educational test must of necessity be adapted 
to generally recognized educational objectives, and to the 
typical or average course of study. Hence, such a test may 
not meet the teacher’s need. Onthe other hand, an exam- 
ination may be constructed by the teacher so as to fit the 
instruction which the class has been receiving. 

It should also be borne in mind that standardized educa- 
tional tests are not fundamentally different from examina- 
tions. In both cases, the pupil is asked to give a performance 
in response to certain exercises, and then this performance 
is described by means of a score or “grade.” , Standard- 
ized educational tests are essentially nothing more than re- 
fined and improved examinations. Hence, the requirements 
of test construction should suggest ways in which examina- 
tions may be improved. Of course, it will not be possible 
for the teacher to follow out in detail the procedure of con~ 
structing standardized educational tests, but there are cer- 
tain phases of test construction which may be incorporated 
in a crude way in the preparation of examinations,’ Because 
examinations will doubtless always be widely used for 
measuring the achievement of pupils, we shall consider cer- 
tain ways of improving them. 

Making examinations more objective. The most serious 
limitation of the ordinary examination is its subjectivity. 
In the first place, it is subjective in its administration. The 
questions are usually written upon the board, and the pupils 
are required to read them. This practice introduces differ- 
ences from pupil to pupil. The light may not be good for 
some pupils; others may have difficulty in reading at a dis- 
tance. Where it is convenient to do so, the administration 
of examinations can be made more objective by supplying 
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each pupil with a typewritten or mimeographed copy of the 
questions. Occasionally it may be wise to dictate the ques- 
tions, rather than to write them on the board. 

The questions should be stated so that all pupils will in- 
terpret them in the same way. [If all pupils do not attach 
the same meaning to a question, the performances which 
they give will not be truthfully indicative of their abilities. 
In the construction of standardized educational tests, some- 
times exercises that are formulated with care are later found 
to be ambiguous. A striking illustration occurs in Form 7 
of the Army Alpha Intelligence Test. The exercise asks the 
one taking the test to tell whether “cleave” and “split” 
mean the same or opposite. Both answers are correct, be- 
cause “cleave”? has two meanings which are exactly op- 
posite. Hence, performances on this exercise are not valid 
indices of general intelligence. Apparently this ambiguity 
was not detected, although the test was prepared with care 
by competent persons. However, in most cases teachers 
will be able to avoid ambiguous exercises if they subject 
their questions to a careful scrutiny with reference to this 
characteristic. 

The pupils should be given very definite instructions con- 
cerning the method of work to be employed. They should 
be told whether they are to work rapidly or slowly. In 
questions in which they are asked to “discuss,” the com- 
pleteness expected in the discussion of the exercise should 
be indicated. They should be given some indication of the 
amount of time they will have for the examination, and 
whether they should answer the questions in order, or not. 
Other items of procedure in which pupils are inclined to 
vary should be specified. 

Increasing the objectivity of the marking of examination 
papers by definite rules. The scoring of examination papers 
has been shown to be highly subjective. There are two ways 
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in which the subjectivity of the scoring can be greatly de- 
creased. If it is possible to have the examination papers 
scored independently by two or more teachers, the average 
of the “grades” assigned to a paper will be more objective 
than the “grade” assigned by one teacher. The marking of 
examination papers within a school system can also be made 
more objective by formulating definite rules. These rules 
may include some illustrations of the types of answers to be 
counted as correct, and of those to be counted as wrong. If 
partial credits are to be given, the rules should include a 
statement of the types of answers which are to be accepted 
for the various partial credits. If credit is to be given for 
correct principle, agreement should be reached concerning 
the rule to be followed. 

F. J. Kelly ! describes an experiment which is indicative of 
the increase in the objectivity of marking of examination 
papers that may be expected from the adoption of uniform 
rules. Six fifth-grade teachers gave the same examination 
in arithmetic to their pupils. Each teacher marked the 
papers for her own pupils, but did not record the marks on 
the papers. The superintendent then asked a teacher, who 
was unusually systematic in marking examination papers, to 
prepare a set of rules to be followed in the marking of these 
papers. After she had done so, she marked all of the papers 
in accordance with this plan. Then the teachers who had 
first marked the papers marked them a second time, fol- 
lowing her plan. This provided two marks by the class- 
room teacher for each paper, the first without following any 
systematic plan, and the second given in accordance with 
the rules formulated. Each of these marks was compared 
with the mark given by the teacher (called the “judge”’) 
who marked all of the papers. 


1 Kelly, F. J. Teachers’ Marks, p. 83. Teachers College Contributions 
to Education, No. 66. 
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In Table XI, the six teachers are designated by the letters 
A, B, C, D, E, and F. The table is read as follows: When 


TABLE XI. DISTRIBUTIONS OF DEVIATIONS FROM A 
“STANDARD ” MARK OF TWO SETS OF TEACHERS’ MARKS 
ON FIFTH-GRADE ARITHMETIC PAPERS — FIRST, WITH- 
OUT ANY EFFORT TO UNIFY THE METHODS USED, AND 
SECOND, BY A COMMON STANDARD (AFTER KELLY) 


Without standard With standard 
Range of 
i ferences 

Ai|B|C|D|E\ F |Tota| A|BiC|D|E| F | Totat 
21 or more 4 2 
16 to 20 1 1 1 3 
152+ 2 2 
14 1 1 
13 1 2 8 
12 1 1 2 
11 1 1 2 4 1 1 
10 1 1 1 1 
9 1 2 1 4 
8 1 3 1 5 
rf 1 1 1 1 1 5 1 1 
6 2 1 1 4 
5 1 2 1 1 2 vf 
4 2 2 1 1 2 10 1 1 2 
$ 4| 2 1 £41 -2 11 1 1 u 3 
2 2 1 1 1 1 8 ae | 1 Se te 17 
1 Bil toAe A Sifesd. 14 18 SCS ao Bay y 1 16 
0 1 4 4 1 1 1 12 | 22} $0] 16 | 16 | 29 | 26 | 139 
1 2 5 2 2 2 1 14 5 2 2 1 8 13 
V4 6 1 3 2 & 1 16 i 1 3 5 
3 9 4 4 13 2 2 1 1 6 
4 5 I 4 1 5 1 17 2 3 3 8 
5 2 3 2 2 1 10 1 1 2 4 
6 1 1 3 2 v4 1 1 4 
if 1 1 6 1 9 
8 g 1 2 1 6 
9 1 1 2 4 
10 1 1 1 3 
11 1 2 
12 1 1 a 3 
18 1 4 A ’ 
14 1 
15 1 1 2 1 1 
16 to 20 g 2 
21 or more Lo eSaprck 5 
Totals 35 | 41 | 35 | 36 | 39 | 83 | 219 | $5 | 41 | 35 | 36 | 39 | 33 | 219 
Medians +3] 0 }+1/+6/-1|—-4] +1 
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no rules were followed teacher A marked one paper 16 to 20 
points lower than the “judge,” one paper 7 points lower, two 
papers 4 points lower, two papers 2 points lower, agreed 
with the “judge” on one paper, etc. The differences be- 
tween the marks given when the classroom teachers followed 
no rules and when they followed the rules as formulated are 
very striking. In the first instance, the marks assigned by 
the teacher agreed with those assigned by the “judge” in 
only 5.5 per cent of the cases, while in the second instance 
they agreed in 63.5 per cent of the cases. This indicates a 
very marked increase in the objectivity of the marking of 
the papers. 

Increasing the objectivity of examinations by using ques- 
tions which permit of only one correct answer. The mark- 
ing of examination papers is subjective largely because the 
scorer is asked to exercise Judgment in determining the credit 
to be given for the pupil’s performance. In spelling, a pupil’s 
performance is either right or wrong, and our practice is to 
allow no credit for a performance which is not entirely cor- 
rect. Thus, the marking of an examination paper in spell- 
ing is highly objective. A high degree of objectivity may 
also be obtained in the operations of arithmetic by agreeing 
_ to give no credit for examples partly correct. In other sub- 
ject-matter fields we are accustomed to ask a few questions - 
which call for specific facts, and hence admit of only one 
correct answer. 

It has been claimed that such questions appeal only to the 
pupil’s memory, and that they do not yield an index of his 
acquaintance with principles and of his ability to organize 
and apply his knewledge. In order to reach this phase of 
his education we have asked the pupil to “discuss,” “‘tell 
why,” “compare,” etc. When a pupil is asked to formulate 
an answer consisting of one or more sentences it is difficult 
or impossible to classify the performance as either right or 
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wrong. When scorers are asked to exercise judgment in 
evaluating such performances, wide differences of opinion 
will be found to exist. In order to overcome this subjectiv- 
ity of marking, it has recently been proposed that we meas- 
ure a pupil’s acquaintance with principles and ideas by 
means of certain types of exercises which permit of only one 
correct answer. ‘These types of exercises have been used in 
our standardized educational tests, and it is now suggested 
that they be used by teachers in the examinations which 
they set. Four types of such exercises will be considered 
below. 

True-false exercises. Instead of asking the pupil to for- 
mulate an answer in response to a question, it has been pro- 
posed that we ask him to tell whether a given statement is 
true or false. For example, instead of asking the pupil, 
“Why did the Puritans come to America in the seventeenth 
century?” we may ask him to tell whether the following 
statement is true or false, “The Puritans came to America 
in the seventeenth century seeking wealth.” The pupil may 
give his answer to this true-false exercise by writing a plus 
sign if he considers it true, and a minus sign if he considers 
it false. The mental processes required in answering such 
exercises do not appear to be the same as those which occur ~ 
in answering questions of the usual type. However, experi- 
mental evidence indicates that there is a very high corre- 
lation between the scores which pupils make on a true-false 
examination, and their acquaintance with ideas and prin- 
ciples as determined by our ordinary examinations. 

In constructing true-false exercises, one may prepare a 
list of statements which cover, in some detail, the portion of 
the subject on which the pupils are to be examined. After 
such a list has been prepared, some of the statements can 
easily be changed to false ones so that the number of true 
statements will approximate the number of false ones. The 
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untruth of a statement should not be too obvious, or it 
will be worthless for testing. An effort should be made to 
secure statements which will require an acquaintance with 
the subject in order to determine their truth or falsity. The 
exercises should be arranged so that there is no regular 
sequence between true statements and false statements. 
Since the pupil can give his responses very quickly, the exam- 
ination should consist of not less than fifty statements. 
A true-false examination of one hundred statements can be 
given in the time usually devoted to an ordinary examination. 

Procedure in the true-false examination. The examina- 
tion should be mimeographed or printed so that each pupil 
will have a copy. He may give his answers in the margin 
of the sheets, or, if it is desired to use the same set of papers 
with another group of pupils, he may be given a sheet of 
paper on which there are numbered blanks. The pupils will 
then be asked to record in the blanks their answers to the 
corresponding exercises. A less desirable plan, which may 
be followed when it is not possible to secure mimeographed 
copies of the examination, is to read the statements to the 
pupils and have them record their answers in numbered 
blanks. The disadvantage of this plan is that the pupils 
would not have a satisfactory opportunity to study the 
statements. There is also a chance that the class may give 
some indication of the answer if a statement should appeal 
to them as being ridiculous. 

The pupils should be given specific directions in regard 
to answering the exercises when they are uncertain. One 
writer! has suggested that the pupils be instructed to guess, 
if they were uncertain concerning the truth or falsity of the 
statement. Another writer * who has used this type of exam- 

1 McCall, W. A. ‘A New Kind of School Examination’; in Journal of 
Educational Research, vol. 1, pp. 38-46 (January, 1920). 


? Wood, Ben D. “The Measurement of College Work”; in Educational 
Administration and Supervision, vol. vu, pp. 801-34 (September, 1920). 


THE IMPROVEMENT OF EXAMINATIONS 285 


ination instructed the pupils as follows: “First, go thfough 
the list quickly and mark all that you know for certain, then 
go back and study out the harder ones. Do not guess; the 
chances are against you on guessing. Don’t endanger your 
score by gambling on those questions about which you 
know nothing.” This is probably the better procedure. 

Since only two responses are possible, it is obvious that a 
pupil may give a correct response as the result of chance. In 
order to take this possibility into account, a pupil’s score 
on an examination of this type is the number of exercises 
answered correctly minus the number answered incorrectly. 
Exercises not attempted are not counted. 

“Yes” and “no” exercises. These exercises are just 
what their name implies. Each exercise is answered cor- 
rectly by either “yes” or “no.” No other answers are ap- 
propriate. The administration and scoring are similar 
to that of the “true-false” exercises. In fact, they may be 
considered as a special case of the “true-false” exercises. 

Recognition exercises. Exercises in which the pupil is 
asked to choose from a number of proposed answers have 
also been used to make the scoring objective. This type of 
question has been called the “recognition exercise.” It has 
been used in standardized silent reading tests, and in a num- 
ber of our group intelligence tests. It may be illustrated by 
the following: 

“The first president of the United States was: Christopher 
Columbus, Benjamin Franklin, George Washington, Thomas 
Jefferson.” 

A pupil may indicate the correct answer by underlining it, 
or marking it in some other way. If only one of the answers 
proposed may be considered correct the marking of such 
exercises will be highly objective. 

Examinations of this type should be mimeographed or 
printed, and a copy given toeach pupil. He should be given 
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definite instructions concerning the method of work to be 
followed. It is probably best to instruct him to work 
through the test rapidly, answering those exercises which 
he is certain he knows. He should then go back over the list 
and try the more difficult ones. Not fewer than four pro- 
posed answers should be given. When this is done the 
chances are slight that a pupil will give the correct answer 
by guessing. The pupils may be instructed to guess if they 
do not know, since the chance of success by guessing is slight. 
The pupil’s score on an examination of this type may be 
taken as the number of exercises done correctly. 

Completion exercises. In this form of test the pupils 
may be asked to fill in the words omitted from statements. 
The following illustrates this type of exercise: 


! Revenue for paying the war debts of the states after the Revolu- 
tionary War was provided by a and by due largely 
to influence. 

The slavery question in such states as should be carved from 
the Louisiana Territory was temporarily settled in by the 


The scoring of completion exercises is not as highly objec- 
tive as the two types mentioned above. Pupils will write a 
variety of words in the blanks. Different words may have 
almost the same meaning, and frequently the scorer will be 
compelled to determine whether the meaning of some word 
is sufficiently near to that of the correct answer to justify 
giving the pupil credit for having answered the exercise 
correctly. In using completion exercises it is necessary to 
provide each pupil with a mimeographed or printed copy of 
the examination. The pupil’s score is the number of blanks 
filled in correctly. 

Other advantages of the “new examination.” In addi- 
tion to increasing the objectivity of the marking of the ex- 
amination papers, certain other advantages of the “new 
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examination” may be noted. There will be a large saving 
of time for both pupil and teacher. The pupil is called upon 
to do little or no writing in giving his answers, and he is, 
therefore, able to respond to a large number of exercises. In 
scoring there will be little or no occasion to exercise judg- 
ment, and the scorer will need only to note the brief responses 
given by the pupils. In consequence the labor of scoring 
will be greatly reduced. The saving of time in the giving 
and scoring will more than offset any additional time that 
may be expended in the construction of the “new examina- 
tion.” 

Another advantage is that the examination can be made 
more comprehensive. It is traditional for examinations to 
consist of ten questions. A few are limited to a smaller 
number, and only occasionally do we find examinations con- 
sisting of more than ten questions. The pupils cannot write 
upon a large number of questions in the time allowed. In 
consequence the scope of traditional examinations is neces- 
sarily narrow. We have stated that true-false examinations 
should include not fewer than fifty exercises. Examinations 
consisting of completion exercises or recognition exercises 
should have a corresponding length. Thus, the “new ex- 
amination” may be made distinctly more comprehensive 
than the traditional examination. 

Limitations of the “new examination.” It does not appear 
likely that the “new examination,” consisting of the types of 
exercises we have described, will entirely replace the tradi- 
tional type of examination. The “new examination” can- 
not be used in mathematics, except to a limited extent. It 
cannot be used at all in English composition. The following 
questions taken from Hahn’s Scale for Measuring the Ability 
of Children in History appear to require mental processes 
distinctly different from those the “new examination” calls 
for. 
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" State points of similarity between the position of the United 
States in 1812 and her position in 1912. 

Arrange the following events in order of cause and effect: Force 
Bill, the Carpetbaggers, Fifteenth Amendment, Negro Rule in 
Some of the Southern States, Ku Klux Klan. 

Name the Presidents of the United States since 1892. 


Furthermore, it is likely that pupils would miss valuable 
experience and training if they were not asked at times to 
compare, explain, discuss, or define. This is also true of 
questions in which they are asked to summarize material 
presented on a topic, or to apply certain principles that have 
been presented. Hence it is difficult to conceive of the “new 
examination” being a complete substitute for the traditional 
examination. 

The unequal difficulty of questions not a serious defect. 
It does not appear that accurate measurements of the abil- 
ities of pupils are secured by giving the same credit for an- 
swering an easy question as for answering a difficult one. 
However, investigations of this question, in connection with 
the scoring of standardized educational tests, have indicated 
that the errors introduced by this procedure, which appears 
to be illogical, are not large. After having weighted the 
exercises of his language and grammar test on the basis of 
difficulty, Charters dropped the weights because he found 
that the correlation between the weighted and unweighted 
scores was slightly over .90.! 

A number of other test-makers have likewise used exer- 
cises which were unequal in difficulty without assigning any 
weighted credits to them. A number of other tests which 
consist of exercises arranged in ascending order of difficulty 
have been scored by taking the number of exercises done 
correctly, which amounts to giving as much credit for doing 


? Charters, W. W. “Construction of a Language and Grammar Scale”; 
in Journal of Educational Research, vol. 1, pp. 249-58 (April, 1920). 
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an easy exercise as for doing a more difficult one. A recent 
study ' of methods of describing the performances of pupils 
indicates that the errors introduced by giving the same credit 
for all exercises are frequently smaller than the variable errors 
of measurement. Although the unequal difficulty of the 
exercises undoubtedly introduces an appreciable error in a 
number of instances, it appears that this source of error, 
particularly when the examination consists of as many as 
twenty or more questions, does not cause gross inaccuracies 
in the examination “grades.” Hence, it is probably safe to 
say that, if extreme differences in difficulty are avoided, and 
an examination consists of as many as twenty questions, one 
may safely neglect the errors produced by the unequal diffi- 
culties of the questions. 

Agreement with minimum essentials. It is obvious that 
the content of an examination should be in agreement with 
recognized educational objectives. The questions should 
be representative of what the pupils should know. Studies 
of minimum essentials made in connection with the construc- 
tion of educational tests or otherwise may be utilized by 
teachers in setting examinations. Such studies as that of 
Ayres in spelling and Hahn in history afford the teachers 
valuable assistance in basing an examination on the mini- 
mum essentials. Catch questions, or questions reflecting 
the hobbies of a teacher, have no place in examinations. 
Neither should questions be asked just because they are 
difficult, or because the teacher believes the pupils cannot 
answer them. The only justification for asking questions 
is that the questions are worth while. 

Recognition of significant dimensions. In considering 
the construction of educational tests it has been pointed out 


1 Monroe, Walter S. ‘‘The Description of the Performances of Pupils 
on Exercises of Varying Difficulty,” in School and Society, vol. xv, pp. 
341-44 (March 25, 1922). See also pp. 128-29. © 
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that the significant dimensions of ability should be meas- 
ured. In the traditional examination, pupils are generally 
allowed all the time they desire or are graded only upon the 
questions that they have time to try, which amounts to the 
same thing. This means that only the quality of the work 
is described. In some subject-matter fields, such as opera- 
tions of arithmetic or handwriting, it is important to have 
the rate of work described. By setting examinations suffi- 
ciently long so that no pupil will finish in the time allowed, 
and by timing the pupils, it is possible to secure a measure 
of their rate of work. Measurement in terms of the level of 
difficulty reached cannot be accomplished except by means 
of standardized educational tests. Fortunately, the meas- 
urement of this dimension of ability is not important in most 
school subjects. 

Norms for ordinary examinations subjective. It is cus- 
tomary to describe the performance of a pupil on an examina- 
tion in terms of the per cent of the questions which he has 
answered correctly. School marks are also generally ex- 
pressed in per cents, or in terms of symbols which are de- 
fined in terms of per cents. Passing marks vary; sometimes 
they are as low as 50 or 60; sometimes one is chosen as 
high as 75 or 80. The fixing of the passing mark at any 
per cent such as 60, 70, or 75 is arbitrary. Usually no dis- 
tinction is made between “‘scores” in terms of per cent, and 
“grades” or school marks. If a pupil answers 72 per cent 
of the questions of an examination correctly he is given a 
mark or “grade” of 72 per cent. The same passing mark 
applies to all examinations and to all types of scoring. 
Thus, the norm or passing mark for a particular examina- 
tion is largely determined by the difficulty of the examina- 
tion and the plan of marking which the teacher follows. 
Hence, it is subjective. A low “grade” may be due either 
to the pupil’s lack of ability or to the high norm set by the 
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teacher as a result of the difficulty of the examination and 
the severity of his scoring. Likewise, a high “grade” may 
be due either to the pupil’s exceptional ability or to an easy 
examination. 

“Grades” vs. measures of achievement. This distinc- 
tion between “grades” and measures of achievement is 
important. Teachers have not generally been aware of it. 
One reason for the failure of teachers to recognize this dis- 
tinction is the fact that both school “grades” and the de- 
scriptions of the performances of pupils are generally ex- 
pressed in terms of per cents. The score which is used to 
describe a pupil’s performance should be distinguished from 
his “‘grade.”” That these are different may be shown by the 
following illustration. 

Suppose that when expressed in terms of per cents the five 
highest scores for an average class of forty pupils are as fol- 
low: 80, 77, 75, 74, 72, and the five lowest are: 49, 47, 46, 44, 
41. Suppose, also, that in this school the “grades” are re- 
ported in per cents, and that 75 is the passing mark. Since 
this class is assumed to be typical or average in ability, it is 
obvious that only a few pupils should receive “grades” 
below passing. Only three of the scores are up to or above 
the passing mark. Hence, it is obvious that these scores are 
different from “grades.” They must be translated into 
school marks, but before this can be done the basis of trans- 
lation must be determined. 

Translating achievement quotients into school marks. In 
the case of standardized educational tests scores. may be 
translated into school marks by comparing them with ap- 
proximate norms and with the standard distributions of 
scores. The general procedure may be illustrated by the 
plan which has been proposed for interpreting the achieve- 
ment quotients yielded by the Illinois Examination." When 


1 Monroe, Walter S. The Illinois Examination, Bulletin No. 6, Bureau 
of Educational Research, University of Illinois. 
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the achievement quotients for the different school grades 
were assembled together it was found that the distributions 
were approximately equivalent. Therefore, it was possible 
to make a single statement with reference to the basis for 
translating all achievement quotients into school marks. 
The distribution was divided, so that a certain per cent of 
the pupils fell into each group. This division might have 
been made in a number of different ways. The one given 
below seemed to be the most satisfactory in the case of this 
particular battery of tests. Instead of descriptive terms, let- 
ters or other symbols may be used. For most purposes five 
groups will be found satisfactory. 


Quality of pupil’s achievement —_ Achvevement Per cent of 
Quotient pupils included 

Very iauperior 6.2.5 sisi Boe j 165 and above ' 1 
135-164 6 
SUDETION che os <tc kt aoe od aachs 117-134 13 
IAVETADCscr. cre 's:c oe ais 8 so cis eres 83-116 60 
PGORers sea cee cet coe ee coos 71-82 13 
Palle... Nits See PAY 55-70 6 
Below 55 i 1 


Translating point scores into school marks. The pro- 
cedure for translating point scores! into school marks is 
similar. It is impossible for a teacher to obtain a standard 
distribution for an examination in the same way we have ob- 
tained standard distributions for our standardized educa- 
tional tests. The examination which a teacher prepares is 
given only to one group of pupils, or at most only within a 
school system. However, crude standards may be deter- 
mined by making use of the fact that the scores of a large 


1 The term “point scores” is used here to include all descriptions of per- 
formances, whether in terms of per cents or in terms of other units. The 
procedure described applies to point scores yielded by standardized educa- 
tional tests as well as to examination scores. 
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number of unselected pupils approximate the normal dis- 
tribution. It will, of course, frequently happen that a 
teacher does not have a typical class. It may be that the 
class is composed of bright children, or that there has been 
an accumulation of dull children in this particular class. In 
any class of ordinary size a close agreement with the normal 
distribution cannot be expected, although when the class 
consists of twenty or more pupils we generally find that the 
distribution of measures yielded by a standardized educa- 
tional test resembles the normal distribution. 

A teacher should first determine whether his class is typi- 
cal or not. The giving of a general intelligence test will be 
helpful in this connection. A distribution of their I.Q.’s 
may be considered a very reliable index of the composition 
of the group. If the median I.Q. of a class is below 100 the 
teacher may know that he has poor pupil material. If the 
median I.Q. is above 100 he may know that the class con- 
sists of better pupils than the average. If there is a rela- 
tively large number of low I.Q.’s it may be expected that 
there will be an unusually large number of low “grades.” 
Thus, by means of the intelligence quotient and in other 
ways, the teacher may come to know the general status of 
his class. 


58 
56 69 
47 55 69 


85 44 55 68 75 

@7 34 42 52 63 74 

25 32 40 50 60 70 
The point scores of a class should be assembled, as shown 
above. There are 23 pupils in this class. The median point 
score is 55. If the class is an average one this median score 
of 55 should be translated into the median or average grade 
which the school recognizes. If the grades are reported in 


994 THEORY OF EDUCATIONAL MEASUREMENTS | 


terms of per cents and the passing mark is 75, the average 
grade will usually be approximately 85. If the class is 
known to possess superior ability the median score of 55 
should be translated into a higher grade. On the other hand, 
if the class is known to be decidedly below average in ability, 
55 should be translated into a lower grade, perhaps as low as 
77 or 78. In an extreme case it might even be translated 
into the passing grade of 75. The translation of the me- 
dian score into a grade furnishes a basis for translating the 
other scores. 

In general, the lowest scores will be translated into grades 
below passing. The per cent of pupils who receive grades 
below passing will naturally vary widely with different 
classes. There is a somewhat prevalent opinion that the 
normal probability curve fixes the per cent of pupils who, 
in the long run, should fail to receive a passing mark. This 
is a mistaken notion. The normal probability curve tells us 
nothing concerning the per cent of pupils who should receive 
any grade until we have determined, from some independent 
source, the range of ability which we desire each grade to 
represent. This is a matter of school policy. The school 
should determine the range of ability to be represented by 
each mark. The best way to tell this is in terms of the per 
cents of pupils who, in the long run, will receive each mark. 
A survey of the present practice would probably reveal that 
for elementary schools and high schools the per cent of pu- 
pils who are failed, varies greatly from system to system. In 
some cases the per cent is high; in others it is low. It also 
varies for different school subjects. 

Objective norms for examinations. A teacher can intro- 
duce a large degree of objectivity in the norms used in in- 
terpreting examination scores. The procedure which we 
have suggested above for translating point scores into school 
grades, and the definition of the different grades in terms of 
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the per cent of pupils who are to receive each grade, will 
tend to make the norms objective. Of course, there will still 
be a large subjective factor, but gross injustice will be pre- 
vented when the teacher happens to set a hard examination, 
and pupils will be given more appropriate grades when the 
examination happens to be unusually easy. 


QUESTIONS AND TOPICS FOR INVESTIGATION 


1. In what ways is an examination different from standardized educa- 
tional tests? 

2. Construct a “‘true-false examination,” and show how you would 
demonstrate its usefulness. 

3. Can we expect that eventually examinations will be completely 
replaced by standardized educational tests? Why? 

4. Are we justified in asking teachers to spend much time in the prepara- 
tion and use of examinations? Why? 

5. Should pupils be excused from final examinations? State your reasons. 

6. How would you determine the reliability of an examination? Can you 
find any account of this having been done? 

7. Secure the scores for an examination that has been given to a large 
number of pupils, and devise a plan for translating these scores into 
school marks. " 


CHAPTER XII 


STATISTICAL METHODS: 
THE FREQUENCY DISTRIBUTION AND ITS 
DESCRIPTION 


Purposes of statistical methods. Statistical methods are 
procedures which are helpful in analyzing and summarizing 
collections of quantitative facts, and the relationships ex- 
isting between sets of paired quantitative facts. A small 
group of facts may sometimes be summarized by crude and 
unconventional methods, but when one is dealing with sev- 
eral hundred facts, or, as is the case of some investigations, 
several thousand facts, it is necessary to follow procedures 
which experience has proved to be effective in dealing with 
such collections of facts. In addition, the use of “‘standard” 
procedures will generally result in an economy of time. It 
may happen that in some particular investigation a differ- 
ent procedure can be used advantageously, but due caution 
should always be exercised when departing from tried meth- 
ods. On the other hand, any procedure which assists in re- 
vealing the true significance of the facts should be used 
unless some recognized method will accomplish the same 
purpose. There is no virtue in involved and intricate pro- 
cedures unless they prove more helpful than simpler meth- 
ods. Other things being equal one should always use the 
simplest methods that are available. 

In order that one’s work may be easily understood by 
others it is advisable to use recognized statistical methods as 
far as possible. Certain procedures are so generally helpful 
that for many of them definite rules have been agreed upon 
and expressed in terms of formule. When these procedures 
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are effective they should be used, and it is best to adhere, 
even in minor details, to the formule which are generally 
accepted. Failure to do so will not necessarily make our 
work incorrect. It will, however, frequently result in our 
being misunderstood, and in discouraging other workers from 
reading accounts of our work because of the peculiar sta- 
tistical methods employed. 

Knowledge of significance of derived facts necessary. 
One may use the formula which defines a statistical proce- 
dure without understanding the reasons for it, or how it has 
been derived. Most users of statistical methods are prob- 
ably unacquainted with the derivation of some of the for- 
mule which they use. It is, however, imperative that one 
understand how to interpret the derived facts (median, 
average, standard deviation, coefficient of correlation, etc.) 
which these formule yield. As has been pointed out above 
it is the purpose of statistical methods to assist in the analy- 
sis and interpretation of quantitative facts. Hence, if the 
significance of the derived measures is not understood, no 
progress in the analysis and interpretation of the original 
facts has been made. A derived fact sustains a definite 
relationship to the original facts upon which it is based. 
This relationship is in the nature of a summary or general- 
ization, but there are certain limitations which must be 
borne in mind. 

Sources of error in the derived facts. The original facts 
are generally not absolutely accurate. This is particularly 
true when they are scores yielded by educational tests. If 
the original facts involve errors certain derived facts also 
will involve errors. In some cases these errors are less for 
the derived facts, but always due consideration must be 
given to them. 

Many studies are made in an attempt to arrive at the 
general laws, or relationships, which exist between two quan- 
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tities, as, for example, the height and weight of children, or 
achievement in Latin and achievement in English, or gen- 
eral intelligence and achievement in school subjects. Since 
it is impossible to gather all of the existing facts with refer- 
ence to such problems, it is necessary that they be studied 
on the basis of a sample of the total number of facts. Even 
when a large number of facts have been collected in a ran- 
dom way they may not be entirely representative. It is, 
therefore, necessary to make allowance for the errors in our 
derived facts which may be due to the use of a sample. 

It is also necessary to bear in mind that, although the 
facts may be collected according to a procedure which one 
might expect to give a random samplying, they may not be 
entirely representative because of the existence of unex- 
pected conditions.! For example, the average or median 
score is used as the grade norm for educational tests. This 
average or median is based upon a large number of scores 
of pupils belonging to a given grade. It is assumed that 
these scores are representative if they have been secured 
from a large number of cities and the total number of scores 
amounts to several thousand in each grade. If these scores 
are obtained from a first trial they will, in general, not be 
representative of the scores that would be obtained from 
the second or third trial of the same test. If they are ob- 
tained from schools in which semi-annual promotion pre- 
vails they may not be representative of scores which would 
be obtained from schools which have annual promotion. 
Again, if they have been obtained from schools in which one 
type of course of study is used, they are not likely to be repre- 
sentative of schools which follow a different course of study. 
lf they are obtained from city schools they will probably not 
be representative of rural schools. 

1 See “ Psychological Examining in the United States Army”; in Memoirs 
of the National Academy of Sciences, vol. xv, pp. 553 ff., for an illustration of 


the difficulties met in securing a representative sample. 
. 
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Plan of treatment to be followed. The consideration of 
the general statistical methods used in the construction, 
administration, and study of educational tests may be con- 
veniently divided into three sections, (a) grouping facts in 
frequency distributions, (0) description and use of frequency 
distributions, (c) relationship between sets of paired facts. 

Grouping facts in frequency distributions. In attempt- 
ing to interpret or summarize a collection of quantitative 
facts it is helpful to arrange them in a convenient order. 
If they are arranged in order of increasing magnitude they 
can be more easily interpreted than when the arrangement 
isarandom one. For example, in Table XII, the scores for 
rate of silent reading for 81 seventh-grade pupils are given. 
If these scores were arranged in the order of increasing mag- 
nitude their meaning could be more easily grasped. 

However, when one is dealing with as large a number of 
facts as those given in Table XII it is helpful to group simi- 


TABLE XII. SILENT READING RATE SCORES OF 81 SEVENTH- 
GRADE PUPILS AS DETERMINED BY STARCH’S READING 
TEST NO. 6 


108 106 220 94 100 194 246 162 50 150 
172 »=«156 94 150 150 100° 220 286 168 426 
106 =158 192 194 460 294 84 220 46 160 
108 = 176 300 260 476 160 194 220 194 220 
56 =220 340 150 108 308 166 134 248 160 
194 134 246 100 194 286 238 246 220 158 
268 178 338 194 316 226 220 194 248 226 
72 84 220 134 148 152 194 252 220 338 


lar facts. In this illustration the facts extend from one score 
of 46 words per minute up to a score of 476 words per min- 
ute. It is difficult to grasp the significance of such an ex- 
tended array of facts. If the scale is divided into appro- 
priate intervals, and the number of facts falling in each in- 
terval is given, we have a summary of the facts which is 
more convenient to work with and more easily understood 
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than the original array. In Table XIII the scale from zero 
to 499 has been divided into intervals of twenty-five units 
each. These intervals are given in the first column. In the 
second column the scores falling in each interval are given. 


TABLE XIII. SCORES GIVEN IN TABLE XII GROUPED IN IN- 
TERVALS TO FORM A FREQUENCY DISTRIBUTION 


Fine Scores Frequency 
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Total 81 


The third column gives the number, or frequency, of scores 
in each interval. A table consisting of the first and third 
columns of Table XIII is called a frequency distribution. 
It consists of, first, a statement of the scale intervals, and 
second, a statement of the number, or frequency, of facts in 
each interval. Usually the total is entered at the bottom. 
It is customary to have the zero, or the smallest unit of the 
scale at the bottom of a frequency distribution table, as in 
Table XIII. . This, however, is a convention and not essen- 
tial. It is recommended, though, that this plan be followed 
because it represents the best statistical practice, and fur- 
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thermore it is in agreement with certain mathematical con- 
ventions. 

When a quantitative fact is placed in a frequency distribu- 
tion it loses its identity. All that we know about it then is 
that it falls somewhere within the interval. Its true value 
may place it at either extremity, or anywhere between. 
The frequency distribution tells us nothing concerning the 
precise value of the fact. In Table XIII there are three 
quantitative facts in the interval from 300 to 324. These 
are 300, 308, and 316. In the frequency distribution we 
know only that three facts are within this interval. We do 
not know their precise values. The facts which are grouped 
within an interval may have different values, as in the case 
of this particular interval, or they may all have the same 
value, as in the case of the ten facts in the interval from 200 
to 224. The frequency distribution is a summary, and de- 
tails naturally are omitted. 

Continuous and discrete series. It is necessary to rec- 
ognize certain characteristics of the facts to which we apply 
statistical methods. Most of the facts with which we deal 
in educational research are quantities in a continuous series. 

A continuous series is one which is subject to infinite 
sub-division. A quantity in a continuous series may be ex- 
pressed to any degree of precision desired. For example, 
theoretically a pupil’s ability to do addition examples might 
be expressed by a score of 8.79. A pupil’s rate of reading 
might be expressed as 2.385 words per second. When we are 
dealing with quantities in a continuous series, if the number 
of facts is sufficiently large, and each is expressed with a 
sufficient degree of precision, there will be no “gaps”’ be- 
tween successive facts when they are arranged in ascend- 
ing order of magnitude.! This condition is seldom, if ever, 


1 Theoretically, irrational numbers are required in order to fill the “gaps” 
in a series. © 
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realized in actual practice, because the number of facts 
collected is always finite and they are not expressed with a 
sufficiently high degree of precision. 

A discrete series is one in which there are “gaps.” For 
example, if our collection of facts consists of the number of 
pupils taught by one teacher, we cannot have fractional 
parts of pupils. If the number of pupils is greater than 
25 it must be at least 26. There is a “gap” between 25 
pupils and 26 pupils which cannot be closed. If a discrete 
series is divided into a number of intervals, each including 
several of the “gaps,”’ we have, so far as statistical proce- 
dure is concerned, a series which approximates continuity. 
Such a series has been called pseudo-continuous. 

Method of expressing facts in a continuous series. A 
quantity in a continuous series is generally described in 
terms of the limits between which it is located. If these 
limits are close together the fact is expressed with a high 
degree of precision. If these limits are relatively far apart, 
the facts are said to be expressed roughly, or with a low 
degree of precision. In general, more labor is required in 
making measurements with a high degree of precision, and 
if the measures are assembled in frequency distributions the 
precise statements of the facts disappear. For these reasons 
most educational tests are constructed so that the facts are 
expressed in terms of the lower limit of the interval in which 
they fall. Fractional parts of scores are dropped. For 
example, in the Courtis Standard Research Tests, Series B, 
a pupil’s score for the number of examples attempted is the 
number which he has completed. If he is working on the 
tenth example when time is called, and has completed the 
answer with the exception of the last figure, this example is 
not counted in his score. His score is 9. In this case 9 
means 9 examples entirely completed, plus some fraction of 
the tenth example. A pupil who has just barely completed 
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the ninth example receives the same score. In this case a 
score of 9 means 9 examples completed plus no fraction. 
The scores yielded by certain of our measuring instru- 
ments are expressed in terms of the point on the scale to 
which they are closest. For example, a score of 50 on the 
Ayres Handwriting Scale means that the quality of the 
pupil’s handwriting is nearer 50 than it is to either 40 or 60, 
or in other words that it is between 45 and 55. 
Assumptions concerning frequency distributions. In 
describing a frequency distribution it is necessary to make 
certain assumptions concerning the way in which the true 
values of the facts in each interval are distributed. In the 
case of a continuous series the usual assumption is that they 
are uniformly distributed over the interval and that the aver- 
age value of the interval is that of its mid-point. For exam- 
ple, in Table XIII, the average of the facts in the interval 
from 200 to 224 is assumed to be 212.5. It is actually 220, 
or greater, if we consider that the scores are expressed in 
terms of their lower limits. The average of the quantities in 
other intervals in this table will differ in most cases from 
the midpoint of the interval. However, this assumption is 
the best single assumption which can be made to apply to all 
frequency distributions, and, in general, calculations based 
on this assumption give results which closely approximate 
the results that would be obtained if the calculations were 
based upon the actual values of quantitative facts. This is 
particularly true when the number of cases is large. 
Questions arising in forming a frequency distribution. 
In the construction of a frequency distribution it is neces- 
sary first to decide upon the intervals to be used. Two 
questions arise: (1) How many intervals? this includes the 
question of how large the intervals are to be; and (2) What 
are to be the limits of the intervals? 
No fixed rule can be given concerning the number of in- 
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tervals which should be used in forming a frequency dis- 
tribution. The labor of constructing the frequency dis- 
tribution, and of making statistical computations on it, will 
be lessened by reducing the number of intervals. However, 
if the size of the intervals is increased the facts grouped 
together will differ more widely. Since a fact loses its iden- 
tity when it is grouped in a frequency distribution, increas- 
ing the size of the interval tends to increase the error in 
derived measures. It is, therefore, necessary to avoid re- 
ducing the number of intervals so that significant errors are 
introduced in the derived measures, and also to avoid a large 
number of intervals, which would increase the labor of mak- 
ing statistical computations without at the same time ma- 
terially increasing the accuracy of the derived measures. 
In general, it is not wise to have the number of intervals 
exceed 20 nor be less than 10. 

In deciding upon the number of intervals it is also wise to 
bear in mind the convenience of the intervals chosen. The 
scale of intervals should be such that the labor of assembling 
the facts in a frequency distribution is reduced to a mini- 
mum. So faras possible, one should choose the intervals in 
agreement with our decimal number system. Intervals of 
5, 10, 20, and 25 are much more convenient to use than in- 
tervals of 6, 13, 16, 21, etc. In general the intervals should 
be equal. This, however, is not imperative, but statistical 
computations are more difficult when intervals are unequal. 
In a few cases unequal intervals are introduced by the pecu- 
liar nature of the scale in terms of which the facts are de- 
scribed. For example, accuracy is usually expressed in 
terms of a percent. If we count both zero and 100, which 
we must do if such measures are included in our collection, 
we have a scale which cannot be divided into equal intervals. 
If the scale of intervals, 0-9, 10-19,......... 80-89, 90- 
99, is chosen, we still have to provide for 100 percent. This 
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is essentially not an interval, but a point on the accuracy 
scale. In the class record sheet for the Courtis Standard 
Research Tests in Arithmetic, Series B, the scale of intervals 
is still further complicated by taking the first interval from 
zero to 49. The Nassau County Supplement to the Hille- 
gas Scale yields measures in terms of an irregular scale. 
The values of the compositions of the scale are 1.1, 1.9, 2.8, 
3.8, 5.0, etc. A score of 2.8 (or as it is sometimes expressed 
3.0) really means that the value of the composition is be- 
tween 2.35 and 3.3. If the intervals are expressed in terms 
of their end points they should be as follows for this scale: 
.55-1.4, 1.5-2.34, 2.35-3.2, 3.34.3, 4.4-5.4, 5.5-6.5, ete. 

The exact limits of the intervals should be specified or, 
if they are not expressed, they should be definitely under- 
stood. The form used in Table XIII is recommended. The 
interval is supposed to begin with the first number, and to 
extend up to and including the second number. If the facts 
being assembled in the frequency distribution are expressed 
in fractional form, it is customary to indicate that any fact 
greater than the second number used in describing the inter- 
val but less than the first of the next interval be included in 
the first interval. This is indicated as follows: 300.0- 
324.9. 

In order to facilitate the publication of frequency dis- 
tributions the intervals are frequently described in terms of 
their lower limits. This would mean, in Table XIII, that 
only the first number would be given, and it would be under- 
stood that the interval extends from this lower limit up to 
but not including the lower limit of the next interval. 

Shape of frequency distribution to be expected. Most 
measures of mental and physical traits form distributions 
which tend to approximate a standard shape when repre- 
sented graphically. A typical distribution is one in which 
there are a few small measures and a few large ones, with 
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most of the measures near the center of the distribution. 
The distribution tends to be symmetrical. Such a distribu- 
tion is called normal. In Fig. 11 each of the three curves 
represents a frequency distribution of several thousand 
pupils according to mental age, as measured by the Illinois 
General Intelligence Scale. These curves closely approxi- 
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mate the normal shape, and are typical of distributions 
formed by accurate measurements of mental and physical 
traits of unselected groups of pupils. The shape of a fre- 
quency distribution depends in part upon the scale of in- 
tervals which has been chosen. A different scale may pro- 
duce a frequency distribution which differs rather strik- 
ingly. For example, in Table XITI the scale of intervals is 
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0-24, 25-49, 50-74, etc. If instead we use an interval of 20, 
as in Table XIV, a distinctly different frequency of distri- 
bution is obtained. Theoretically, this should not happen, 
but practically it does. It is probably caused by the pres- 
ence of certain constant errors in some of the rate scores, 
due to certain peculiarities of the test. 


TABLE XIV. FREQUENCY DISTRIBUTION OF FACTS IN 
TABLE XIII, GROUPED ACCORDING TO A DIFFERENT 
SCALE OF INTERVALS 


itaoas Scores Frequency 


460- 460, 476 2 
440-459 2 
420-439 | 426 1 


340 1 
320-339 | 338, 3388 2 
300-319 300, 308, 316 3 
280-299 | 286, 294, 286 3 
260-279 | 260, 268 2 
240-259 | 246, 248, 246, 246, 248, Zax + 6 
220-239 | 220, 220, 220, 220, 220, 220, 238, 220, 226, 220, 226, 222, 220 13 


180-199 | 194, 192, 194, 194, 194, 194, 194, 194, 194, 194 10 

160-179 | 162, 172, 168, 160, 176, 160, 166, 160, 178, 176 10 

140-159 | 150, 156, 150, 150, 158, 150, 158, 148, 152 

120-139 | 134, 134, 184 9 

100-119 | 108, 106, 100, 100, 106, 108, 108, 100 8 

80-— 99 94, 94, 84, 84 4 
0 3 


40- ms 50, 46, 56 


Description of a frequency distribution. A frequency 
distribution represents the first step in summarizing a col- 
lection of facts. It in turn may be described in terms of 
(1) its shape, (2) its central tendency, and (3) its variability 
or spread. 

The shape of a frequency distribution. The usual shape 
of a frequency distribution is one which approaches that of 
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the normal-frequency curve. A normal-frequency distribu- 
tion is completely described in terms of its central tendency 
and its variability. Therefore, we may secure a significant 
description of a frequency distribution by showing the ex- 
tent to which it departs from the normal shape. This can 
be shown graphically by superposing upon the graphical 
representation of the distribution the normal-frequency 
curve having the same central tendency and variability. 

One way in which a frequency distribution may depart 
from the normal shape is being un-symmetrical. This type 
of irregularity is called “skewness.” Formulas have been 
proposed for obtaining a numerical statement of the amount 
of skewness. Thorndike approves the formula,” 

pone ee (mean — median) 
c 

The central tendency of a frequency distribution. The 
central tendency of a frequency distribution is just what the 
term implies. It is the “center” of it. Three character- 
istics of the frequency distribution are used as the central 
tendency; (1) average,* (2) median, and (3) mode. Other 
measures of central tendency could be defined. In fact the 
harmonic mean is used for certain purposes. 

The average. The average is a familiar term. It has 
long been used in connection with school marks, and even 
pupils in the elementary schools understand how to compute 
their “average grade.”” This is done by adding together the 


1See H. O. Rugg, Statistical Methods Applied to Education, pp. 210-13, 
for the method of doing this. 

* See G. Udney Yule, An Introduction to the Theory of Statistics, p. 150, 
for formula in terms of the quartile deviation. 

3 Average is used here instead of “arithmetical mean,” which is pre- 
ferred by some authorities. One reason given for the use of arithmetical 
mean is that the word “‘average”’ is used with a general meaning to include 
all central tendencies, instead of just one. However, average is properly 
used in this more restricted sense. , 


’ 
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different grades received, and dividing by the number of 
such grades. The quotient is the average. This method is 
convenient to use when the number of facts to be averaged 
is small, but it is a cumbersome procedure when the number 
is large. It is also not a convenient method to use in con- 
nection with a frequency distribution. The method de- 
scribed below is one to be used with frequency distributions. 
It is sometimes spoken of as the “short”? method. 
Calculation by the “short method.” The procedure for 
calculating the average is illustrated in Table XV. The 
frequency distributions is taken from Table XIII. The first 
step in applying this method is to assume an average. This 
may be chosen at any point, but the arithmetical calculation 
will be reduced to a minimum when the assumed average 
is chosen at the mid-point of the interval in which the true 
average falls. If it is chosen elsewhere the same results 
will be obtained, but the labor of calculation will be slightly 
increased. In applying this method it is necessary to bear 
in mind the assumption that the average of the measures 
within a scale-interval is considered to be at its mid-point. 
In Table XV the average has been assumed to fall at the 
mid-point of the interval from 200 to 224, or at 212.5. The 
eight scores in the interval 225 to 249 are 25 units, or one 
scale interval above this assumed average. In order to 
reduce the calculations to a minimum the deviation is called 
one scale-interval. In the same way the deviations of the 
other scores from the assumed average are expressed in 
terms of intervals. A negative deviation means that the 
scores fall in an interval below the assumed average. The 
deviations are given in the third column of the table. In 
the fourth column of the table the products of the deviations 
and frequencies are recorded. The sum of the positive 
products is 80, and the sum of the negative ones is —132. 
The difference is -52. This is divided by the total of the 
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frequencies, which gives a quotient of -.642. This means 
that the assumed average is too large by .642 of an interval. 
Since the interval is 25 units, this quotient is multiplied by 
25 to find the correction in terms of units. This correction 
subtracted from the assumed average gives the true average 
of 196.45. 


TABLE XV. ILLUSTRATING THE CALCULATION OF THE 
AVERAGE BY THE SHORT METHOD © 


Frequen 
Scale- Deviation a re 
intervals Frequency in intervals Deviation 
(f) (fd) 
475 1 11 11 
450 1 10 10 
425 aI 9 9 
400 8 
875 x 
350 6 
825 3 5 15 
300 8 4 12 
275 8 3 9 
250 3 gz 6 
225 8 1 8 
200 10 0 80 
175 18 —l —13 
150 15 -—2 —30 
125 4 —3 —12 
100 8 —4 —32 
75 4 —5 —20 
50 3 —6 —-18 
25 1 7 = 
0 ~—182 
Total 81 4 


Assumed average ... .212.5 
Correction. ..... —16.05 


True average...... 196.45 
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The average calculated according to the above method 
will not be exactly equivalent to the average obtained by 
dividing the sum of the scores by their number. The sum 
of the scores from which Table XV was derived (see Table 
XII) is 15,880. This sum divided by 81 gives a quotient of 
196.04. The difference is due to the fact that when the 
scores are assembled in a frequency distribution they are 
assumed to have the value of the mid-point of the interval 
in which they are located. By referring to Table XIII it 
will be seen that this assumed value is not in agreement 
with the actual value, but since the scores must lose their 
identity in the frequency distribution it is necessary to make 
some assumption concerning their magnitude, and this is the 
best single assumption which can be made. 

It has already been pointed out that most scores are ex- 
pressed in terms of a lower limit. Fractions are generally 
dropped. For example, if a score of 9 yielded by Courtis 
Standard Research Tests, Series B, were expressed with a 
high degree of precision it would appear as 9.43, or pos- 
sibly 9.429. When the scale-intervals are only one unit, 
assuming that the scores within any interval are concen- 
trated at its mid-point tends to correct for dropping fractions 
in the scores. In the above illustration we should expect 
the average calculated from the distribution to be slightly 
larger than the one calculated from the scores. It is actually 
0.41 larger (196.45-196.04 = 0.41). In some cases it will 
be smaller, due to the fact that the scores are not uniformly 
distributed. 

Calculation of average when intervals are irregular. If 
the intervals of the frequency distribution are irregular in 
magnitude the calculation of the average is more compli- 
cated, although the above method can be applied. It is 
necessary to express the exact deviation of each interval 
from the assumed average. Table XVI illustrates the cal- 
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culation of the average for one case of irregular intervals. 
The intervals in this table are equal, except the one for 100. 
This is really not an interval. It has no width. The aver- 
age is assumed at the middle of the 50 to 59 interval, or at 
55. The distance of 100 from this assumed average is 45 
units, or 4.5 intervals. When this is used as the deviation 
the calculation of the average offers no difficulty. 


TABLE XVI. SHOWING CALCULATION OF AVERAGE FOR ONE 
CASE OF IRREGULAR INTERVALS 


Frequency 
Scale- Deviation in x 
antervals Frequency intervals Deviation 
(f) (d) (fd) 
100 3 4.5 13.5 
90-99 0 4 
80-89 1 3 3 
70-79 2 2 4 
60-69 5 1 5 
50-59 5 0 25.5 
40-49 4 1 —4 
30-39 3 2 = Gil 
20-29 2 3 —6 
10-19 1 4 —4 
0- 9 —20 
Total 26 
Assumed average... ... 55.0 
Correction. ..... 2.1 
True average...... 57.1 


If the scale of intervals has been chosen so that it is in- 
correct to assume that the measures are concentrated at the 
mid-point of the interval, due allowance for this must be 
made in choosing the assumed average. For example, if 
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the scores obtained from the use of the Ayres Handwriting 
Scale are arranged in a frequency distribution whose inter- 
vals are indicated as 40, 50, 60, etc., the assumed average 
must be chosen as 40, 50, or 60, and not as 45, 55, or 65, be- 
cause as this scale is ordinarily used 40 means from $5 to 44, 
50 means from 45 to 54, etc. 

The median. The median is generally thought of as the 
middle score, or, more specifically, as the score which has as 
many scores above it as it has below it. To guide one in the 
calculation of a median it is necessary to have a more pre- 
cise statement. A recent writer! has given the following 
definition, ““When measures are arranged in order of size 
the median is the middle measure or (lacking a middle 
measure) midway between the two middlemost measures.” 
In applying this general definition to some groups of scores 
which arise in the use of educational tests it is necessary to 
supplement it by certain arbitrary rules. The median may 
be calculated from the measures when they are simply 
arranged in order of size. It may also be calculated from 
the frequency distribution. 

The median of discrete measures. When discrete meas- 
ures are arranged in order of size it is only necessary to lo- 
cate the middle measure, if the number of measures is odd. 
This middle measure is the median. If the number of meas- 
ures is even there is no middle measure. Then, according 
to the above rule, the median is midway between the two 
middlemost measures. ‘This is their average. It is, of 
course, necessary to correct this average for absurdity 
when the scores are discrete. For example, it is absurd to 
give a median in a discrete series as 7.5 children.? The me- 
dian is either 7.0 or 8.0. 

1 McCall, W. A. “How to Compute the Median”; in Teachers College 


Record, vol. xx1, p. 183 (March, 1920). 
2 Although it is logically absurd to use such fractions as 7.5 children, the 
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When discrete measures are assembled in a frequency 
distribution whose intervals are one unit, all of the measures 
are concentrated at the lower limit of the interval. It is, 
therefore, only necessary to locate the interval in which the 
middle measure falls, if there be one. In case there are two 
middlemost measures falling in adjacent intervals, the 
lower limit of either interval may be used as the median. 
In case the two middlemost measures fall within the same 
interval the lower limit of this interval is the median. 

The median of continuous measures. Calculation of the 
median of continuous measures is less simple. If the meas- 
ures are arranged in order of size, it is necessary to estimate 
the most probable value of the middle measure, or, in case 
there is no middle measure, of the two middlemost measures. 
To do this it is necessary to remember that the measures 
are expressed in terms of lower limits. 

For example, the following scores were derived from the 
Courtis Standard Research Tests, Series B:— 2, 3, 4,4, 5, 5, 6, 
6, 6, 6, 7,7, 7,7, 8, 8,9. There are seventeen scores in this 
group. The middle measure is the ninth one. This is the 
third 6. The most probable precise values of the four scores 
which are given as 6 are 6.125, 6.375, 6.625, and 6.875. Thus, 
the value of the middle measure of this distribution is 6.625. 
If another measure is added to the distribution to make 
eighteen there is no middle measure, and we must use the 
two middlemost measures. Suppose that a score of 10 is 
added. The two middlemost measures are then the third 
and fourth 6’s. ‘The median is the average of these, 


6.625 +6.875 _ 
: oe 


practice may be justified for some uses of derived measures. Frequently 
the use of fractions will assist in making comparisons more precise. How- 
ever, the logical absurdity should not be forgotten. 

1 These values are “most probable” according to the assumption of 
uniform distribution within the interval. 


6.75 
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When the two middlemost measures fall in adjacent inter- 
vals the median is the junction point between them. This 
rule is inconsistent with taking the average of the values of 
the two middlemost measures when the frequencies in the 
two intervals are not equal. However, the inconsistency 
is more than compensated for by the ease of calculation 
which it makes possible. 

Calculation of the median from a frequency distribution. 
The calculation of the median from a frequency distribu- 
tion is illustrated in Table XVII. The total of the fre- 


TABLE XVII. ILLUSTRATING THE CALCULATION OF THE 
MEDIAN OF THE FREQUENCY DISTRIBUTION. THE 
SCORES WERE DERIVED FROM COURTIS STANDARD 
RESEARCH TESTS, SERIES B 


Number of Number of 
examples pupils 
DL Ota eten kine clare ats ree cia eta 's ssc ste e 

Aids Ose chleye V aiclbtawelaie Mists minieres ait ae 

UA) Se icerstik pias ils Shee tic sleyseveelsérs 4 

NS cores ssierisle pi elaiets le svete aie S cipon.O ce 5 

MORE ES Sop Jone DUD Ons Se Ummm ooe 5 

a a eR ee his gk ie dle tr pe pe 7 

Deere rete este a eieleteres slate atetole rete sie’e & 

DEA PONE MAUIEASERLRS Saincs's 4 

Beatie hemi dae see tie es veceHo 

Ret iOat Ske fe Ra ole Sd on BB 2 

ise DODO TID OOO SOUL Go nee or 1 

Detetaate chsevaceisie siverers tel sie)ecielsve chee) efess — 

EN rose colts alesis calorie ste e soe © 1 

DRA ae haa te eerd amples eielee Fon 1 

Ce Been bey IA aC PS ORS NOOR — 

1 Foto 6 OOD OO Oo ary so tre Oar ce = 

(0 / Seacictsl emia Gro IC ts » AOSD Or == 

OLA eine st ea a Seg ies uate se ies 40 
Coade Median. 8 5260 sairis oaens 11.00 
CorrectiOR.», «ose 7< > 4 eck ween 


True Median; ......<..s...3. 11.87 
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quencies is 40. To calculate the median this total is di- 
vided by 2, which gives 20, the half sum. We then sum the 
frequencies, beginning at the bottom, until the partial sum 
is such that the addition of the next frequency will give a 
sum greater than 20. In this illustration a partial sum of 16 
is reached by the addition of the frequency in the ten inter- 
val. The addition of 7 in the next interval will give 23, 
which is greater than the half sum 20. The addition of the 
frequency of the ten interval means that we have entirely 
covered this interval. Therefore, the beginning of the 
next interval is taken as the crude median. Thisis11.0. In 
order to secure a partial sum of 20 it is necessary to use four 
of the seven pupils whose scores fall in the eleven interval. 
In other words, we need to use four sevenths of the cases 
in this interval. Since they are assumed to be uniformly 
distributed over the interval, this is equivalent to saying 
that the median is reached when we progress to a point four 
sevenths of the width of the interval above 11.0. Since the 
width of the interval is 1.0, the true median becomes 114 or 
11.57. 

We may also calculate the median by beginning at the top 
of the distribution and counting downward. When this is 
done the addition of the frequency in the 12 interval gives a 
partial sum of 17. Since we are counting downward we have 
progressed to the lower limit of the 12 interval. Hence, the 
crude median is 12.0. To arrive at the true median we need 
three of the seven scores which fall in the eleven interval. 
In other words, we need to progress to a point three sevenths 
of the width of the interval below 12.0. Hence, the true 
median is 12.0 minus #, or 11.57. This is the same result 
as was obtained by counting upward. 

In the above illustration the total of the frequencies was 
an even number. When it is an odd number the procedure 
is exactly the same. Table XVIII illustrates the calculation 
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of the median in such a case. The half sum is 19.5. The 
maximum partial sum counting up is 14, which carries us 
to the beginning of the ten interval. In order to secure a 
partial sum equal to the half sum 5.5 scores within the ten 
interval are needed. Dividing 5.5 by 9, the frequency in 
this interval, a quotient of .6 is obtained. Since the width 
of the interval is 1.0, this is the correction to be added to the 
crude median. Hence, the true median is 10.6. 


TABLE XVIII. ILLUSTRATING THE CALCULATION OF THE 
MEDIAN WHEN THE TOTAL OF THE FREQUENCIES IS AN 
ODD NUMBER. SCORES DERIVED FROM THE COURTIS 
STANDARD RESEARCH TESTS, SERIES B 


Number of Number of 
examples pupils 
1 eae ae oiaibis alerts raleVeuntoisieiersie® witha 1 
BOD ei ctk sc cine tsi clei oie Maas ae 3 
Dees reese ses a mipisteisversiersie em isieie 5 
REFS Meee ne se APSO OIC 2 OSI 8 
NOS. eciees « Rioiste sia Wtiorslens be ete a athte’y 9 
Dire Bia ce cite bie ai tviouhe OE AC TTD 6 
$3 cence urorartions oneiiee are aidieisi che seis ot 4 
fe Se SR Tae SoA pier asieisis Sete k g 
Orie at csieie siaieiecetels th GAS sas 1 
Beocnth ete etd aces stcleieretetcreietehetsts &  siv.cie's —_ 
yi on SE Ra 524 Free poor eas 1 
Sai tak Lebete ole sick ovetiohs Yale stevete pisislels sas —_ 
De ea: Hekate. sheers tareveneteve-aie abelswbsrowse oye — 
a SARs SO RO Ons hae Be eae — 
Oe teas Focsuelevoeigie eae Oo: ehete seis vaneyss —_ 

Potalere ocr otc co ue wee tes a coo 
Crude Median.........+-+-.--- 10.0 
Correction. ......+- Bintan cers 6 
True Median......... Gt ee 1056 


In both of the above illustrations the width of the inter- 
val has been 1.0. When the width of the interval is greater 
than 1.0 it is necessary to multiply the quotient obtained by 
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the width of the interval in order to find the correction. For 
example, in Table XIII, the width of the interval is 25. The 
half sum for this table is 40.5. The maximum partial sum 
counting up is 35. To obtain the half sum it is necessary to 
use 5.5 of the scores falling in the 175 interval. The quo- 
tient, dividing 5.5 by 13, is .4. This quotient must be mul- 
tiplied by 25, the width of the interval which gives 10, the 
correction. Hence, the median is 175 plus 10, or 185. If 
one counts downward in this distribution the maximum par- 
tial sum is. 83 and the required addition is 7.5. The quotient 
is .6 which makes the correction of 15. The median, there- 
fore, is 200 minus 15, or 185. 

Calculation in special cases. Several special cases arise 
which create difficulties. When the maximum partial sum 
obtained in counting upward is equal to the half sum, the 
median is the upper limit of the interval whose frequency 
was last added, or, what is the same thing, the lower limit of 
the next interval. For example, in Table XIX a partial 
sum of 15 is reached by the addition of the frequency in the 
nine interval. Thisis also the halfsum. Hence, the median 
is 10.0. If we count downward we use the lower limit of 
the last interval whose frequency is added. 

It may occasionally happen that the two middlemost 
measures fall on opposite sides of a gap in a distribution. 
Table XX illustrates this condition. The total of the fre- 
quencies is 90. A half sum of 45 is obtained by counting 
upward to the 180 interval. It is also obtained by count- 
ing downward to the 180 interval. In this case the median 
is the average of the upper and lower limits of this interval, 
190 and 180, or 185.! 

The presence of irregular intervals in a distribution do 
not affect the calculation of the median unless it happens to 
fall in an irregular interval. Even in such cases it is only 


‘1 This statement is not precisely accurate. See page 314. 
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necessary to remember the width of this interval. A pecu- 
liar condition which frequently arises in dealing with the 
scores as tabulated on the class record sheet in the Courtis 
Standard Research Tests, Series B, is shown in Table X XI. 
In this case the first frequency is greater than the half sum 
10. Hence the crude median is zero, and the correction is 
ten elevenths of the width of this interval, which is 50. 
This gives a correction of 45.5, which is the true median. 
TABLE XIX. ILLUSTRATING CALCULATION OF MEDIAN 


WHEN IT FALLS AT THE JUNCTION POINT OF TWO IN- 
TERVALS 


Number of Number of 
examples pupils 
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err ere eee Staak «Ame hss dat 5 
Mh iseasherees egos Sars EESG RN vaets wae 3 
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Calculation of percentile points in a distribution. In calcu- 
lating a median we are simply finding a point in a distribu- 
tion such that 50 per cent of the measures are on either side. 
It is possible to calculate the point which represents any 
division of the frequency distribution. For example, we 
may calculate the point which has 25 per cent of the meas- 
ures below it, and 75 per cent above it. Such a point is 
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TABLE XX. ILLUSTRATING THE CALCULATION OF THE: 
‘MEDIAN WHEN IT FALLS IN A GAP. RATE SCORES ON 
STARCH’S SILENT READING TEST NO. 7, GRADE VII 


Words per Number of 
minute pupils 
SOO soe kas. Wee Meas c stenctaterstarete states 1 
S80 A eo eR —- 
+ | Ure ee aes a BMPR Ar Pens APach — 
SEO YS hac Cacaterdy onthe Sho eens shane daustetare — 
SOV he ete. Cae ait tie cea _— 
S40 Si Reet, ee ee eae ettaes — 
SSO Se er ee alae ae Se So chetake atone — 
SRO ters Sercla hye ota ee Oe ee 1 
DLO sw dere spaveieta Sip were a8 Save cea ares — 
SOON. Sse croiarcie aleve detecaein eliccoraie.ceet elas —_ 
DOO Te acute ansiae 0.8 atewioaaste antertastentrare 1 
BBO Rais hao ee ee ee Cane ee i 
STOSE RES Peas Ieee ee 3 
QED in Baa e Tire ee alts a te rae — 
PHO Ani oier ae wiete his Sk aeRO Ree 1 
21 RED, ARTIC RTA eocs aeRO RG —_ 
B30 noms, there wre bigness Cate ee 2 
BEE SAY Neal an ie 8 tae i otis CSA og no 
BLO EE Wren wre Se state ere nee 6 
ROO Sere ere aaa SIRS ere ee 1g 
O05 Siar corset aioe aatobe ke Saree 14 
TEGO sets a eer apace com ee catenins —— 
ONG os mrccarceictoace a Saree Rertareny lakers ere 6 
TOO oes cs tte ac otis ard RT ETE ears 5 
POOR cae cee a Ga eee ee eee 8 
40%. cee oe vcs Ge He eee g 
MSOs ervertetes ates taneieleneimencterieereiates 4 
QO Sed occ tien cit eitreetits @hfeenrts ...14 
DUO srederers he aaecortiu te vie heehee ces cLarecrers I 
TOO So ebeg ccs slorale Gen recta teateteretate ere g 

DOME a caotiay v's vs Selo eree he cere ene tees 6 
BO RG Cae ae g 
Total . oct tase. Ge ee 90 


called the 25-percentile. The calculation follows exactly 
the same procedure as that for the median, with the excep- 
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tion of dividing the total by 2 to find the half sum. To find 
the 25-percentile we divided by 4 to find one fourth of the 
sum. The 75-percentile is the point which has 75 per cent of 


TABLE XXI. COURTIS ARITHMETIC TESTS, SERIES B, 
DIVISION. PER CENT OF ACCURACY 


Per cent of Number of 

aitempts correct pupils 
NOOSE SEs Pa. Sti ted ets Bape S 3 
GOT OD reiiascts ia-c¥s «ster oravorve ws AG Sone == 
PE nan vip al + & eo a6 5's 8 siaak odes 3 
CDOs hia tees os mums 6) 6 56 Ree.0 5% 1 
GO=CO went eee se tc cee ee ste te cs = 
DOSS OME S ears carta bie sees scle ee cee 2 
DEAD tare cite aio viele sic olin eisai vinie eels 11 
(otal westes ox cheek uh wets i5ibe> a5 20 


the measures below it, and 25 per cent above it. It is calcu- 
lated by taking three fourths, or 75 per cent of the total, in 
the place of the half sum. 

Similarly, we may calculate any percentile point, 5-per- 
centile, 10-percentile, 17-percentile, etc. In all cases the 
procedure is identical with that for the calculation of a me- 
dian, except instead of finding the half sum, which is really 
the 50-percentile sum, we find the per cent of the total of the 
frequencies corresponding to the percentile point desired. 

Mode. The mode of a frequency distribution is the value 
of the interval which contains the largest frequency. No 
calculations are required in the determination of a mode. 
It is only necessary to examine the frequency distribution 
and locate the interval in which the maximum frequency 
occurs. Some statisticians have attempted to calculate a 
theoretical mode, which is defined as the mode in the corre- 
sponding frequency distribution of an infinite number of 
cases and infinitely small intervals.! 


1See Rugg, H. O. Statistical Methods Applied to Education, pp. 100-08, 
Also, Whipple, G. M. Manual of Mental and Physical Tests, pp. 19-20. 
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Measurement of variability of a frequency distribution. 
The variability of a frequency distribution refers to the 
deviation, or spread of the measures, about the central tend- 
ency. Some frequency distributions are spread out very 
widely about the central tendency, while others are closely 
grouped about it. The difference between the central tend- 
ency (the median or average) and a measure is its deviation. 
If the measure is less than the central tendency, the devia- 
tion is negative; if it is greater, it is positive. 

The amount of variability of a frequency distribution is 
measured in several ways. One of the simplest measures is 
the average deviation. It is simply the average of all the 
deviations. This average is taken without regard to sign. 
The method of its calculation is obvious, when the measures 
are not grouped in the form of a frequency distribution. It 
is simply required to find the sum of the deviations, and 
divide this by the number of measures. 

Calculating average deviation of a frequency distribu- 
tion. When the measures are arranged in a frequency 
distribution the calculation is more difficult. Table XXII 
illustrates the calculation of the average deviation of a fre- 
quency distribution. The median of this distribution is 
11.57. In calculating the average deviation the arithmetical 
work will be greatly lessened if we use, instead of the true 
median, an assumed median at 11.5, the mid-point of the 
median interval. After we have made our calculations it is 
relatively easy to correct for the difference between this 
assumed median and the true median. In the third column 
of the table the deviation of each interval from the assumed 
median is given. The fourth column contains the products 
of the deviations and frequencies. The total to these with- 
out regard to sign is 87. This divided by 40, the total of the 
frequencies, is 2.175. 

This, however, is not the true average deviation because 
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TABLE XXII. ILLUSTRATING THE CALCULATION OF THE 
AVERAGE DEVIATION OF A FREQUENCY DISTRIBUTION 


No. of 
Number of : Deviation 
examples vim (d jd 

16 1 5 5 

15 2 4 6 

14 4 8 12 

13 5 2 10 

12 5 1 6 

ll vi 0 — 

10 4 -—1 —4 

9 4 —-2 —8 

8 3 -3 —9 

6 2 —4 —8 

6 ul —5 —5 

5 _— —6 —~ 

4 1 —7 —7 

3 1 -8 -8 

2g — — _ 

1 oues — _ 

0 pees — — 
Total 40 87 


Median = 11.57 


it is based upon the assumed median rather than the true 
median. To correct for this proceed as follows: — The de- 
viation of the 12 interval was taken as 1. Its true deviation 
is .93. (12.50-11.57. =.93). Hence, we have used the devia- 
tion for this interval which is .07 too large. The same thing 
has been done for all of the intervals above 12. For the 10 
interval the deviation from the assumed median was taken 
as—l. From the true median it is —1.07. In this case the 
difference between the assumed median and the mid-point of 
this interval gave a deviation which was .07 too small. The 
deviations used for all intervals below the tenth are likewise 
.07 too small. 
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This disposes of all the intervals except that in which the 
median falls. The only difficulty which arises in making 
the correction is in regard to this interval. It is to be dis- 
posed of by the following rule. If the true median is above 
the mid-point of the interval it is to be counted with the 
intervals below the median. If the true median falls below 
the mid-point of this interval it is to be counted with the 
intervals above. Since the deviations for all of the intervals 
above the median are in error by the same amount, we may 
find the total of their frequencies by disposing of the median 
interval by the rule just given. In this table there are seven- 
teen frequencies which were given deviations .07 too large. 
There were twenty-three frequencies which were given de- 
viations .07 too small. Stated algebraically, the error in the 
total deviation is equal to 17(-.07) +23(.07). This is equal 
to .42. This, added to 87, makes the true total of the prod- 
ucts (fd) 87.42. The true average deviation is to be ob- 
tained by dividing this total by the total of the frequencies. 
This quotient is 2.18. 

The above procedure may be summarized by the follow- 
ing formula: 


2 fd +c( Nz — Na) 


A.D. = V 


In this formula = fd means the summation of the products 
of the frequencies or deviations without regard to sign. In 
Table XXII it is the sum of the numbers in the column 
headed fd. cis equal to the true median minus the assumed 
median. Ny, is the number of measures below the true 
median. It is assumed that the measures are concentrated 
at the mid-point of the interval. N, is the number of meas- 
ures above the median. N is the total of the frequencies. 
When the width of the interval is greater than one it is nec- 
essary to multiply this quotient by the width of the interval. 
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‘This formula applies only when the correction is equal to or 
less than one interval. 

The calculation of the average deviation has been ex- 
plained with reference to the median. It may also be cal- 
culated with reference to the average. The procedure is 
identical with that described. 

Standard deviation. The variability of a frequency dis- 
tribution may also be measured in terms of the standard 
deviation. ‘This statistical term is generally referred to as 
“sigma” (a). It is defined by the formula 


In this formula d represents the deviation of any interval 
with respect to the true average, f stands for the frequency 
of measures within the interval, and N is the total of the 
frequencies of the distribution. When the width of the in- 
terval is not equal to one it is necessary to multiply the 
square root by the width in order to obtain o in terms of the 
unit of the measures. The standard deviation is used only 
with reference to the average. 

Table XXIII illustrates the calculation of the standard 
deviation from a frequency distribution. The procedure, 
up to a certain point, is identical to that for calculating 
the average.! In the last column of the table the pro- 
ducts of the frequency and the square of the deviation of 
each interval are entered. The total of these products 
form the numerator in the formula for the standard de- 
viation. 

In this table the deviations have been expressed from the 
assumed average 212.5. The true average is 195.45. It is, 
therefore, necessary to correct for the error introduced by 

‘this assumption. This error is —.642 of an interval. The 
1See page 309. 
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TABLE XXIII. ILLUSTRATING THE CALCULATION OF THE 
STANDARD DEVIATION OF A FREQUENCY DISTRIBUTION. 
THE MEASURES ARE RATES OF READING, AS MEASURED 
BY THE STARCH SILENT READING TEST 


Scale- 
intervals (f) d Ja ja* 
475 1 11 ll 121 
450 1 10 10 100 
425 1 9 9 81 
400 8 
375 As 
350 6 
325 3 5 15 75 
300 3 4 12 48 
Q75 3 $8 9 27 
250 8 4 6 12 
225 8 Ht 8 8 
200 10 0 80 : 
175 13 -—1 —13 13 
150 15 —2 —30 60 
125 4 -—3 —12 36 
100 - 8 —4 —32 128 
T5 4 —5 —20 100 
50 3 —6 —18 108 
25 1 —7 — 7 43 
0 
Total 81 —132 966 
a = 4/288 — ae 
81 
=V 11.5188 


= 3.40 
¢ (in terms of units) = 85.0. 


procedure for making the correction is indicated by the 


following formula: 
3 fq? 


N 


In this formula d is the deviation of the mid-point of an 
interval from the assumed average. Since d and c are 
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expressed in terms of intervals, it is necessary to multiply 
the value of sigma obtained from this formula by the width 
of this interval, if it is desired to have the standard deviation 
expressed in terms of the unit. 

Other measures of the variability of a frequency distribu- 
tion. One measure of the variability of a frequency dis- 
tribution is half of the difference between the 75-percentile 
point and the 25-percentile point. This is called the quar- 
tile range. The procedure of calculation is obvious. It is 
simply necessary to find the 75-percentile point and the 25- 
percentile point, subtract the one from the other, and divide 
the difference by 2. 

The median deviation, or probable error (P.E.). In case 
the distribution is approximately normal, the quartile range 
can be interpreted as a measure of deviation from the me- 
dian. It is, thus, essentially the median deviation. This 
measure of deviation has been called the probable error (P. 
E.). When used in this connection the term is somewhat 
misleading, and some authorities advise against its use. 
The median deviation (or P.E.) can be calculated directly 
in case the distribution is approximately normal, but in gen- 
eral the simplest procedure is to derive it from the standard 
deviation. The procedure is indicated by the following 
relationship between these two measures of deviation: 


P.E. = 67450 


This relationship assumes a normal distribution or one 
which deviates from it very slightly. 

Relationship between measures of variability. When a 
distribution is normal the quartile range is the same as the 
median deviation, which is the probable error (P.E.). The 
standard deviation, average deviation, and P.E. are defi- 
nitely related to each other. The various relationships are 
given by the following equations: 
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o= 1.2533 A.D. 
o = 1.4826 P.E. 


A.D.= “82. ¢ 
A.D. = 1.1843 P.E. 
P.EW= 6745 0 
P.E. = .8458 A.D. 


These relationships hold only when the distribution is 
normal, or approximately so. They may be used to obtain 
any two of the three measures of variability when the third 
one is known. 

Interpretation of measures of variability. The median 
deviation, or probable error, permits of the most simple in- 
terpretation. If we think of a distance equal to the prob- 
able error being measured on either side of the average, 
and the points Av.—P.E. and Av. + P.E. being located, fifty 
per cent of the measures included in the distribution will be 
found between these two points. For the other two meas- 
ures of variability the following statements may be made: 

Between Av. — o and Av. + o there are 68.26 per cent of the 
measures in a distribution. 


Between Av. — A.D. and Av. + A.D. there are 57.5 per cent of 
the measures included in the distribution. 


It is possible to extend these statements with reference 
to these measures of variability. However, it is not usually 
done except for P.E. 


Between Av. — 2 P.E. and Av. + 2 P.E. there are 82.26 per cent 
of the measures in the distribution. For a measure chosen at ran- 
dom from a distribution the chances are 4.5 to 1 that the measure 
chosen will lie between these two limits. 

The chances that a measure chosen at random will lie between 
the Av. + 3 P.E. and Av. — 3 P.E. are 21 to 1. The per cent of 
measures included between these two limits is 95.70. 

The chances that a measure chosen at random will lie between 
‘Av. — 4 P.E. and Av. + 4 P.E. are 142 to 1. The per cent of 
measures included between these limits is 99.30. ° 
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The chances that a measure chosen at random will lie between 
Av. — 5 P.E. and Av. + 5 P.E. are 1310 to 1. The per cent of 
measures included between these limits is 99.924. 

The chances that a measure chosen at random will fall between 
Av. — 6 P.E. and Av. + 6 P.E. are 19,200 to 1. The per cent of 
measures falling between these two limits is 99.995. 


The effect of chance errors of measurement upon the 
central tendency and measures of variability. Measures 
involve two types of errors—constant and variable. 
Constant errors affect central tendencies by making them 
larger or smaller by the amount of the error. If the 
constant error becomes known, corrections can easily 
be made in the central tendencies. Constant errors do 
not in any way affect measures of variability... The effect 
of variable errors of measurement on central tendencies 
is decreased as the number of measures is increased. The 


1 , 
variable error of an average equals VN times the variable 


error measurement. Thus in order to reduce the variable 
error of the average by one half it is necessary to increase 
the number of measures four times. 

Measures of variability are increased by the presence 
of variable errors. That is, a distribution of measures 
which include variable errors is spread out more than the 
distribution would be if the variable errors were eliminated. 
The effect of variable errors upon the standard deviation 
is given by the following equation: ” 


SF True scores = F Obtained scores Vr. 


1 Constant errors may be relative or absolute. A relatively constant 
error is one which bears a constant ratio to the measure. This type of 
constant error does affect measures of variability. 

2 Kelley, T. L. “The Measurement of Overlapping”; in Journal of 
Educational Psychology, vol. x, pp. 458-61 (November, 1919). .” 
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In this formula the true scores refer to the scores after 
the variable errors have been eliminated. The obtained 
scores include the variable errors. 7 is the coefficient of 
reliability. Since both the probable error and the average 
deviation can be expressed in terms of the standard 
deviation, it is a simple matter to derive the corresponding 
formule for the other two measures of variability. 

Effect of using a sample upon the central tendency 
and measures of variability. When a central tendency 
or measures of variability are interpreted as expressing 
generalizations, it is necessary to allow for the fact that 
they are calculated from a limited sample of measures. 
Assuming that the sample used was selected without bias, 
there is a chance that another sample selected in the same 
manner would yield a slightly different central tendency 
and measure of variability. It is, therefore, necessary 
to indicate the extent of the error which is introduced by 
this possibility. The formule generally used for this 
purpose are the following: 


oe SD. aistestation 

Pe nas ah a 

2 S-D.dsstesbution 

S) P.E. ian = 8454 ae 
( ) med: Vv N 

S.D. istribution 

(8) P.E-sandard deviation = 6745 = ——— VE v 


Comparison of two averages. In comparing two averages 
it is necessary to take into account the errors which each 
involve. There is a possibility of errors due to one or more 
of three causes — constant errors in the original measures, 
variable errors in the original measures, and sampling. 
The latter source does not apply when the averages are not 

1 See page 238, 
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treated as generalizations. The probable error of the 
difference of two averages due to variable errors of meas- 
urement and due to sampling is given by the formula, 


P.E Difference as See 


averages P.E “averages 

This assumes that the measures whose averages are 
being compared are unrelated. 

The use of measures of deviation. Measures of devia- 
tion or variability have a number of important uses in the 
construction and use of educational tests.2, The standard 
deviation (a) is also used in the calculation of coefficients 
of correlation, as described in the following chapter. 

EXERCISES 

Assemble the following tables of data in frequency distributions. Exer- 
cise care to form the frequency distribution which will give the most useful 
summary of the data. 

1. The numbers below represent a random sampling of the salaries of 
Illinois principals taken from the Illinois School Directory for 1920-21. 
$1650 $1700 $2000 $2000 $2000 $2500 $2000 $3000 $2000 

2000 1650 1600 2300 2800 3000 2600 2700 2000 
2000 1700 2025 2500 2200 2000 1500 1350 2300 
1000 3000 2600 2700 1260 1500 1600 1800 1400 
2259 2800 2400 2000 4500 3200 1800 3037 1500 
2000 3000 1600 2000 2500 1560 2000 4000 2700 
1800 3200 2000 3000 2000 2500 2000 3300 3384 
2750 2250 1700 900 2400 1800 3000 2000 3000 
38000 1700 1650 


2. The following are scores derived from Monroe’s Genera] Survey 
Arithmetic Scale II, Form 1, Grade VIII. 


1See page 209. 
2 See pages 95, 297, for descriptions of a number of these uses, 
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_ 8. Rate scores derived from Monroe’s Standardized Silent Reading Test 
II, Form 2, Grade VII. 


188 141° 10 dis “115° 10% “10” 1625 Ss Ore 152 
188 162 141 10% 162 115 141 162 138 I62 16% 
73 78 152138" 162 162—(«dGZ) 162 | 162 Ge est 
141 162 152 162 115 115 78 101 133 M41 #162 
162 162 141 162 115 73 89 133 162 115 162 
162 162 162 152 4133 #162 OL 162 1385 «162 152 
1441 162 1383 162 133 141 152 141 #4133 «#162 162 
162 133 162 162 162 162 133 162 141 W41 162 


4. The following numbers are intelligence quotients. They represent a 
random sampling of a small city school. 


94 79 102 106 102 105 96 100 84 100 74 
765 103 131 109 102 = 105 97 6101 94 103 123 
105 =108 98 105 115: 114 114 104 100 110 = 105 
103 119 101 124 87 97 90 illo WO 101 180 
101. 130 101 108 79 92 79 105 93 103 83 
92 104 125 105 113 


5. Accuracy scores yielded by the Courtis Standard Research Tests in 
Arithmetic, Series B. Division. 


89 88 71 67 0 91 100 86 88 88 71 62 67 78 67 
73 100 78 80 0 44 100 100 100 7 83 100 78 90 100 
50 73 67 100 80. 57 57 85 89 87 91 90 85 75 100 
87 100 20 83 89 50 82 60 87 89 87 89 25 100 91 
87 738 83 100 100 44 92 50 100 83 78 


6. The numbers below represent the values of compositions written by 
high-school pupils as determined by the Nassau County Supplement to the 
Hillegas Scale. In rating the compositions each was assigned the numerical 
value of the scale composition which it was considered to equal most nearly 
in merit. Thus, a composition which is given a value of 7.2 is considered to 
be more nearly equivalent to the scale composition having this value than it 
is to any other step of the scale. 


5.0 2 


7. GO GOS 5:09 18.8" | $8500 79: 5.0 7 7.2.6.0 
$38) 8.0 G0. 620) F725 3.8, 2:85 6 


0: 1.9 ~ 6.0 
-O S.06- 7.2% 


Calculate the median, average, and standard deviation of the following 
frequency distributions: 
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7. Distribution of scores obtained 
from one algebra test. The 
score is the number of exercises 
solved correctly. 


Number solved Number i) pupils 
US Sy 60 Scacegackiade 
Daal ciel din’s Sine is ae a 
bh ag aags .130 
Uietevars'clavelele rete ctoe 152 
Aa SEG .144 
ease Gagoodoae 118 
Aedes ratte a oleic at 56 
Seca renmcncaad 70 
S65 ROIS .. 34 
Le eee ssansages 183 
ip eran eases « 10 
Number tested.......... 938 


9. Height of male students, Ohio 
State University. (61 means 
nearer 61 than 60 or 62, etc.) 


Inches Number % students 
ANG velieve|aie's/eis\e10,o/e1s\ 9,010.0 
USincietem > vaeieleisie sistatele : 
ib GHC S5oG SOOO ae 
Ph laaotqeaacasedaaoaae. thes 
Eee Bracers SG ae dona tet sh 

Fe OU arate ois in sissies sia niente « 109 
OSes chico ciesewensie > 126 
(Officine aanacacoaaducdor Mg 
GG i cto. aiorelslele)sneimicie 44,99 
Sng po anadsonogosaed08 57 
64..... ete ielecsloks SYysage 38 
OSc hie Soar eonope wl 
G2 aereoieie Gisleinletoire iis 6 ae kU 


GES CIE croie ceive pisieen ee g 


833 


8. Mental ages of pupils tested 


with Illinois Examination, Nov, 
1920. 

M.A Number a pupils 
De Natl mitstee cies Ere eee 

°F Lees SRA IER AT aS 3 
PONS aise heciectetee ok 58 
ON ravers ste a oaioste lesion 3 
WO Mectte erasers ts sisi writer 13 
LV iota crane is Panic 26 
NG cerercitoos sare meio nee 62 
BD eraejerecnn arse ore soi aie ole 146 
BA pin acre eis nsyneoaret mete 212 
MD se reatern cic apen oe crete cies 827 
fC oer acy ORT retanyaaic 431 
Bee ee oe is ae eee 486 
LO oie cave stale wea ais careers 481 
Aros OMS ICIS. ts IACI 480 
Sd Gea cineAc arto 496 
i cdo cap clonidior ensayo 367 
Ger reteis a ercin soc tshunates «iscale © 248 
Dees ae tate elaietoers ss -.- 26 


10. Scores derived from Monroe’s 
General Survey Arithmetic 
Scale II, Form 1, Grade VIII. 


Interval _ Frequency 
OS seis ots ARSnopnaaacod 
AO Natasa ois oc) atelarele's wleleleeteeie 
MOO iemrsiete,eiajs pele cies see, 
ns Saren Se Sactonisoon 2 

a eae dvecaenedece st? 
NUS Aa a Aadric aiteree te. 
MO regs ee win ayaa arate Abpoos scat 
OEE cuss te ae ae cecohejaieisisin ee 
LG SR AC SAS aoa nacee Arar 
POO EG ienu ets iervsicnis wie ats aoe 
BO ii esse she Haas cSassaoos 

Otel, nevus svesescebsegeendl 
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+11. Distribution of a random sam- 
i pling of salaries paid to prin- 
cipals in Illinois. 


Interval Frequency 
Ab OO tricrele tievete eraraicloveleiale aoa 


Wotaletroccchioa sic eases case eu 


13. Accuracy scores yielded by the 
Courtis Standard Research 
Tests in Arithmetic, Series B. 
Division. 


Interval ' Frequency 
LOD vecsaccvesdvescedccsslt 
DO rasecaewewensaserinnw O 
BO Uienecvs ue neat ese see 
WO ns ceave¥ecatsccucaquewkt 
COiavcinvewessenacnrye GO 
OOKian ce emanoien wie nsec 


Totaly ccsvbavs ssbice Coren eee 


12. 


Distribution of intelligence 
quotients. They represent a 
random sampling of a small 
city school. 


Interval Frequency 
ESS Gace caaee ae erect ae 


(WE AGercocotsecenedéich 


Dotali.ccccccccseesncsicele 500 


14. 


Arithmetic Achievement Ages 
of an Illinois County. Tllinois 
Exam. Oct. 1920. 


Number of pupils 
1 


tee eesesreesereere 


see eeeeoececeeesos © 
seeee ees ecoreecece 
see eeeeeernrerseee 
seeoe sere reseeecees 
soeee eerertoreonoeen 
’ 
oY 
seeeee seers reoesee 


cere eeeedeoeeses 


eee eee ee 
seers eoenreoeovrece 86 
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14 continued 15. Scores derived from Bucking- 
ham’s Scale for Problems in 


A.A. * Number of pupils Arith. Form 1, 
ULES Gog Reon Oe 34 
DOs cieyetevejeises,s/s eves pase 84 Score Number of pupils 
G-Or acs Wie s punters 89 DAO = sera state aio’ ss gists eietecaie’s $28 
SHO ea he rales eae 169 BB emcee ars Moab ana ® 782 
Ge oie ccerdlaiparalsasieaie a 237 8 Os cos adie ens saeaters 1349 
MU seit soles ae ote a Oeste 252 SOT trike « sralda siete Reine etn 2931 
GHG ec este steer ene 149 oO sath. core aleibia/cicee here 58 
G-OFF. aon see 34 OC. B= 5. sis wake ceils acu 
CO cote totes inion a tele ste a UBS Se oe See e 
as PaO ee SRO In ToS Ble Ub Dire eccin teeta eyo eae ee ° 
A=Gordeccicig ve sae eno e 3 G5 Os ems eoese ts \trers wre : 
S—-O=S-D criialale « sie ee we! MAD=. a. pide we we ta colsieneis 
Bi Da isis Wis pair aa atvare-ouy ° 
SLOLAI Oe tee aie ats cies 1147 Barn ass cio a enielce sis S 
Sacer cise insetorslete ier aie 
DOH e eels oti alsintoe a piste minis 
0-2.4..... a nae 1184 


CHAPTER XIII 


STATISTICAL METHODS: 
RELATIONSHIP EXISTING BETWEEN SETS OF 
PAIRED FACTS 


Sources of paired facts. Paired facts are secured when two 
tests are given to the same pupils. This would occur when 
an arithmetic test and a silent reading test were given. We 
also have paired facts when a single test yields two scores, 
as, for example, a rate score and a comprehension score in 
the case of silent reading. Any two measures of the same 
pupil constitute a pair of facts. We may thus have paired 
facts resulting from a single test which has two scores, from 
two different tests, or from two applications of the same test. 
A test score may also be compared with the pupil’s chrono- 
logical age, his grade placement, his school marks, or with 
any other measure of his mental or physical traits. There 
are other sources of paired facts, but these are sufficient to 
indicate the need for methods of studying the relnugne 
existing between them. 

Arrangement of paired facts. In studying the stone 
which exists between paired facts it is desirable to arrange 
them in a way which will assist in understanding the nature 
of this relationship. In general, the best arrangement is 
secured by tabulating the facts in the form of a correlation 
table. Sometimes this table is spoken of as a double-entry 
table. The character of the table is illustrated in Table 
XXIV. 

Constructing a correlation table. The first step in con- 
structing a correlation table is to decide upon the class 
intervals. The directions given on page 303 apply here. It 
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is not necessary that the number of intervals be the same 
for both sets of facts. Unless the distributions of the two 
sets of facts are irregular, the choice of intervals will affect 
the coefficient of correlation very little. The procedure to 
be followed in tabulating paired facts in a correlation table 
depends in part upon the form in which the original data 
are available. In most investigations in which correlation 
is used it is desirable to have each pair of facts recorded on 
a separate card. When this has been done, the best method 
of procedure is to sort the cards according to the intervals 
for one set of facts in the same way as they would be sorted 
if a frequency distribution was being formed. After the 
cards have been thus sorted each pile should be taken up in 
turn and sorted according to the other set of facts. The 
number of cards in each pile should then be recorded in the 
correlation table, as shown in Table XXIV, which is for 
rate scores obtained from Form 1 and Form 3 of the Courtis 
Silent Reading Test No. 2. 

When all entries have been made the totals of the columns 
should be recorded in the line marked f at the bottom of the 
table. ‘The totals of the lines should be entered in the col- 
umn marked f at the right of the table. This column at the 
right of the table contains the distribution of the Form 1 
scores. The line f at the bottom contains the distribution 
of the Form 3 scores. The totals of these two distributions 
should be identical. In Table XXIV they are 80. 

Calculation of the coefficient of correlation by the Pear- 
sonian formula. The processes of calculating the coeffi- 
cient of correlation by means of the Pearsonian formula 
may be described in terms of the following steps: 

Step I. Construct a correlation table as described above.! 


1 The coefficient of correlation may be calculated without the con- 
struction of a correlation table. Two methods have been proposed for 
doing this. The most commonly used method is described in Rugg’s 
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TABLE XXIV. CORRELATION TABLE FOR RATE SCORES OBTAINED FROM FORMS 1 
AND 3 OF THE COURTIS SILENT-READING TESTS NO. 2, GRADE IV 


Form 1 


50 +90 | 70 80 | 90/100] 110/120] 130]140]150:1160 
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30 | 40 
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Step II. Estimate the averages of the distributions at the 
mid-point of a convenient interval. Draw horizontal and 


FORM 1 

0 5 10 15 20 25 30/35]40 45 5055 60 65 f d fd fd? XY 
6 18 108 90 
5 1050 30 
4 1248 298 
315 45 39 
22040 26. 
ele ae 
0 88 


fr 25 er) 8-2-1632 38 
O 20 3 ! 4-3-1236 42 
15 3 2 1 6-4-2496 96 
10 2 3—5-I575 60 
5 I I= 66 36 ,'. 36) 
0 77 583 7.492 
f Hi ETa Secor Mill0 16 <9'6.4. Wo anys C,= 2508 = —5)” 
do }6—-5-4~3-2-! 0 | 2 3 4 5 6 x = 2601 
fd 6 -30-28-15-I8—liHi08 6 18 18 16 5 6{60~ c, = Ua. 
fd? 36 150 1l2 45 36 II 6 36 54 64 25 36/61  CI=.0196 
C,Cy=—.0714 
o,= of 5-260 oj= 282-0196 
= 2.79 1207 
922 +0714 
gd TT Nk 


- 


=,828 
Fia. 12. Tae CaLtcuuaTIoN oF THE COEFFICIENT OF CORRELATION BE- 
TWEEN SCORES DERIVED FROM Form 1 AND Form 2 oF THE ILLINOIS 
GENERAL INTELLIGENCE SCALE 


vertical lines marking off these intervals in the correlation 
table, as shown in Fig. 12. Extend the table at the right 


Statistical Methods Applied to Education, pages 274-76. A second method 
has been proposed by Dr. L. P. Ayres, Journal of Educational Research, 


vol. 1, pp. 216-21 (March, 1920). 
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and at the bottom to provide for the following quantities: 
deviation from the assumed average (d), product of each 
deviation by its frequency (fd), and the product of the fre- 
quency and the square of the deviation (fd?). This proce- 
dure is identical with the calculation of the standard devia- 
tions of the two frequency distributions. The standard 
deviation of the horizontal distribution is called o, and the 
standard deviation of the vertical distribution is called o,. 

Step III. The two columns at the extreme right are 
headed =zy. The entries to this heading may be either 
positive or negative. It is customary to provide space for 
two columns, one for the positive entries and the other for 
the negative entries. The procedure for determining the 
entries in these columns is as follows: Each frequency in 
the correlation table is multiplied by the product of its devia- 
tions from both assumed averages. In Fig. 12 there is a 
frequency of 1 in the first line under 55. The deviation of 
this from the assumed average of the vertical distribution 
is 6. The deviation of this from the assumed average of 
the horizontal distribution is 4. This frequency is, there- 
fore, to be multiplied by the product of 6 and 4 or 24. 

Since all of the frequencies in a row have the same devi- 
ations from the assumed average of the vertical distribution, 
the arithmetical work will be reduced if each frequency in 
a row is multiplied by its deviation from the assumed aver- 
age of the horizontal distribution, and the sum of these 
products multiplied by the deviation of the row from the 
assumed average of the vertical distribution. In Fig. 12 
the sum of the products of the frequencies and their devia- 
tions from the assumed average of the horizontal distribu- 
tion for the “65 row” is (4 X1)+(5 X1I) + (6 X1) = 15. 
This multiplied by 6 gives the total amount of 90, which is 
the entry to be made in the =zy column. The other entries 
in this column are determined in a like manner. 


; 
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It is necessary to bear in mind the sign of the deviation. 
For example, the entries on the “40 line” are partly to the 
left of the assumed average and partly to the right. The 
first frequency is 2. Its deviation is -2. The product is 4. 
The next frequency is 1 with a deviation of -1. This gives a 
product of -1. The frequency 4 has a deviation of 0, hence 
the product is 0. The first 3 has a deviation of 1, and 
the second 3 a deviation of 2. The sum of these products is 
—4—1+3-+6=4. This, multiplied by 1, the devia- 
tion of the row from the assumed average of the vertical dis- 
tribution, gives the product of 4 which is entered in the last 
column of the figure. Sometimes the product will be nega- 
tive. In such cases the entries are made in the “minus” 
column. 

Step IV. The coefficient of correlation is now calculated 
by the formula: 


OzFy 


In obtaining Szy we take the algebraic sum of the products. 
In Fig. 12 all of them are positive. If some were negative 
the sum would be the difference between the positive and 
negative products. In Fig. 12 the product of ¢,c, = —.0714. 
Since this negative product is to be subtracted it means the 
addition of a positive quantity. The calculation of the 
coefficient of correlation is completely illustrated in Fig. 12. 
This arrangement of the work is recommended. 
Interpretation of the coefficient of correlation. The gen- 
eral appearance of a correlation table suggests the type of 
relationship which exists between the two sets of quanti- 
ties recorded in it. If all the entries fall upon the diagonal 
line, as is shown in Table XXV, there is a perfect relation- 
ship or correspondence'between the two quantities. In this 
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illustration the two paired facts are the cost of an amount 
of coal and the number of tons, the price remaining constant. 
The table is to be read as follows: One lot of 2 tons costs 
$20, four lots of 4 tons each cost $40 each, eight lots of 6 
tons each cost $60 each, and soon. If we know the number 
of tons of coal, the cost of that amount is definitely deter- 
mined. If we pass in one set of facts from small to larger 
quantities, the paired facts also increase regularly, and al- 
ways in the same ratio. In case the entries fall upon the 
other diagonal line of the table there is a definite relationship 
between the two sets of facts. However, this relationship 
is inverse. When one fact is large the other is small. The 
coefficient of correlation is a numerical index of the relation- 
ship between the two sets of paired facts. When all the en- 
tries in the correlation table fall upon the diagonal line, as 
in Table X XV, the coefficient is + 1.00. If the entries all fall 
upon the opposite line of the table the coefficient is —1.00. 


TABLE XXV. ILLUSTRATING PERFECT CORRELATION. THE 
RELATION BETWEEN THE COST OF VARIOUS AMOUNTS OF 
COAL AND THE AMOUNT 


Amount in tons 


— |—— | ——_— | |} ————_ | — | ——— | — |] | | LL, 


80 10 
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.If we measure the weight and height of a number of 
children we have two sets of paired facts. For each child 
there are two facts — his height and weight. We know 
from general observation that, in general, tall children 
weigh more than short ones. We also know that there are 
numerous exceptions to this rule. Some relatively short 
children weigh more than others who are taller. We may 
describe the relation between the height and weight of chil- 
dren by saying that there is a tendency for tall children to 
weigh more than short ones, although this relationship is 
not perfect. Hence the coefficient of correlation for these 
two sets of facts will be less than 1.00. 

If twelve dice are thrown one hundred times and the num- 
ber of 4’s is recorded for each throw, the two sets of paired 
facts formed by pairing together the odd numbered throws 
and the immediately following even numbered throws are 
not in any way related except by chance. For example, if 
an odd numbered throw resulted in eight 4’s, no prediction 
can be made concerning the probable number of 4’s which 
will appear on the next even numbered throw. There is 
absolutely no connection between these two sets of paired 
facts. If these paired facts were recorded in a correlation 
table they would be scattered over the entire table in a mis- 
cellaneous order. The coefficient of correlation for such a 
table would be approximately zero. Secrist ! gives a corre- 
lation table of 500 pairs of throws of twelve dice. The 
coefficient of correlation is r = .04 + 03.., 

t Facts obtained in mental measurements do not exhibit 
perfect relationship except by chance. Correlation tables 
are not obtained in which all the entries fall upon a diagonal 
line. Frequently they are grouped near this line, but in other 
cases they are scattered over most of the correlation table. 


1Secrist, Horace, An Introduction to Statistical Methode, p. 484, Mace 
millan, 1917. 
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A few concrete illustrations of correlation may be help- 
ful. The coefficient of correlation of the height of fathers 
with the height of sons has been calculated to be .51. For 
the age of husband with the age of wife it is .91. The num- 
ber of mothers’ children with the number of daughters’ 
children gives a coefficient of correlation of .21. The corre- 
lation between success in mathematics and success in Eng- 
lish has been found to be .40. Reliability coefficients of 
most educational tests fall between .60 and .90.! 

Effect of variable errors and constant errors upon the 
coefficient of correlation. The presence of absolute con- 
stant errors in either one or both sets of paired facts has no 
effect whatever upon the coefficient of correlation. Relative 
constant errors may have a slight effect. The presence of 
variable errors always tends to lower the coefficient of corre- 
lation. When two measurements of each of the two traits 
which are being studied are available it is possible to cor- 
rect for the effect of the presence of these errors by certain 
formule.? This is called correction for attenuation. 

The effect of sampling upon the coefficient of correlation. 
In the study of paired facts we may seek to ascertain if any 
relation exists between the traits of which we have a limited 
number of measures. For example, we may ask if there is 
any relation between achievement in Latin and achieve- 
ment in English, or achievement in arithmetic and achieve- 
ment in silent reading. The usual procedure in studying 
such questions is to secure, for a number of representative 
pupils, measures of their achievements in the two fields under 
consideration. ‘These measures may be the scores obtained 
from standardized tests, or they may be the school marks 
received in the two school subjects. The coefficient of corre- 


1See page 203. 
2 Thorndike, E. L. An Introduction to the Theory of Mental and Social 
Measurements. Teachers College, Columbia University, 1916, p. 179. 
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lation is an index of the relation which exists between such 
sets of paired facts. 

However, in interpreting it is necessary to bear in mind 
that a coefficient of correlation is necessarily computed 
from the facts gathered. These are generally only a sample 
of all of the existing facts. In a study of the relation of 
achievement in Latin to achievement in English, measure- 
ments of these achievements would be secured from only 
a few of the total number of pupils. The study would 
be unusually comprehensive if several hundred cases were 
included. We are, therefore, endeavoring to determine a 
general relationship upon the basis of a sample. For this 
reason it is necessary to make due allowance for the possi- 
bility of the sample not being completely representative of 
the total, even though it was chosen in an unbiased or ran- 
dom manner. 

The addition of other groups of facts to our correlation 
table may change its general appearance, and the coefficient 
of correlation calculated from it. In fact, this will happen, 
unless the additional facts are related to each other in ex- 
actly the same way as the facts in our first table. Hence, 
the coefficient of correlation calculated from one group of 
measures of the traits would likely differ slightly from that 
calculated from another group of measures. The average of 
all such coefficients of correlation is taken as the true index 
of the relationship. The difference of any coefficient from 
this average is called the “error due to sampling.” The 
magnitude of the error of a coefficient of correlation, due to 
sampling, may be described in terms of the limit which is 
exceeded by 50 per cent of the coefficients of correlation 
calculated from similar collections of these paired facts. 
This limit is the probable error. 

Computing P.E. of the coefficient of correlation, due to 
sampling. The probable error of the coefficient of correla- 


346 THEORY OF EDUCATIONAL MEASUREMENTS, 


tion, due to sampling, can be computed by the following 
formula: + 


The meaning of the probable error of the coefficient of corre- 
lation may be illustrated as follows: If a coefficient of .20 
has been obtained from a correlation table based on 36 cases; 
i.e., 36 pairs of facts, the probable error of this coefficient of 
correlation is .11. This means that the chances are 1 to | 
that the true coefficient lies between .09 and .31. The 
chances are 4.5 to 1 that it lies between — .02 and .42. The 
chances are 21 to 1 that it lies between — .13 and .53. 

It is obvious that we are not justified in asserting that the 
true coefficient of correlation — i.e., the one calculated from 
all of the pairs of facts, or the average of the coefficients cal- 
culated from a large number of collections of 36 pairs of 
facts — is positive. Therefore, a coefficient of correlation 
of .20 with a probable error of + .11 cannot be considered 
proof that a relationship exists between the two traits. Al- 
though the calculated coefficient is positive, the probable 
error is so large in comparison with the coefficient of corre- 
lation that it is not altogether certain that the true coeffi- 
cient would be positive. For this reason, it is necessary to 
require that the coefficient be several times larger than 
its probable error before it is interpreted as proving the 
existence of a relationship. Since the probable error 
of a coefficient of correlation is frequently needed in its 
interpretation, it is customary to express them together 
asr = 20 + 11. 

Authorities differ with reference to the ratio which must 
exist between the coefficient of correlation and its probable 
error before the existence of a relationship can be asserted. 


1 Tables have been prepared from which the value of P.E. can be read 
for various values of 7 and N. ; 
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By at least one writer the ratio is placed at 6. Another 
writer places it as low as 2 or 3. A conservative rule is that 
the coefficient must be four times its probable error before 
the existence of a relationship can be assumed. 

What a relationship between measures means. Proof 
that relationship exists between two quantities means only 
that they tend to change together. If one quantity becomes 
larger there is a tendency for the other also to become larger 
in case the relationship is direct. If it is inverse, as one 
quantity becomes larger there is a tendency for the other to 
become smaller. The larger the coefficient of correlation, 
the more marked is the relationship. For the relation of 
achievement of one school subject to that of another school 
subject, most of the coefficients of correlation range from 
.30 to .50. Thus, .60 is considered to indicate a high degree 
of correlation for these two traits. This only means that 
.60 is larger than most of the coefficients of correlation ob- 
tained in the study of the relationship existing between the 
achievement in different school subjects. In the study of the 
reliability of educational tests, coefficients of correlation 
from .60 to .90 are most frequent when the scores are secured 


from pupils belonging to a single grade. Hence, for this 


relationship, .90 is considered high and .60 is low. 

The establishment of the existence of a relationship be- 
tween two traits does not mean that one is the cause of the 
other. For example, there is a positive relationship between 
achievement in Latin and achievement in English, but that 
does not necessarily mean that success in Latin is the cause 
of success in English. It may be that success in both sub- 
jects is dependent upon a third factor, such as general in- 
telligence. The interpretation of coefficients of correlation 
requires an intimate acquaintance with the traits which are 
being studied, and in many cases with other traits. 


Is 
é 


» 


The departure from perfect correlation. In studying the 
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objectivity and reliability of educational tests we are not 
concerned with establishing the existence of a relationship 
between the two sets of scores. The presumption in favor 
of a relationship is so strong that its existence can in general 
be assumed. In most cases it is marked. Our interest is 
rather in knowing the amount of departure from complete 
or perfect correlation. The coefficient of correlation in- 
dicates this only in a very general way. It is, therefore, an 
unsatisfactory index of the objectivity and reliability of edu- 
cational tests. The probable error of estimate is proposed as 
a more significant index. In order to understand its mean- 
ing it is necessary to consider first the regression equation. 
The regression equation. The general relation between 
two sets of paired facts may be represented by the regression 
equations: 
y =r ov 5 eae y (A) 
Oz Oy 
In these equations z and y are deviations from their respec- 
tive averages, 0, and o, are the standard deviations, and 
r is the coefficient of correlation existing between the two 
sets of facts. If y’ represents the absolute magnitude of the 
y-facts, and zx’ the absolute magnitude of the z-facts, and 
a, and ay are the averages of the two distributions, we may 
write these equations in terms of them as follows: 


(y'— a) =r * (e/a) (w'-a,)=r= y'-a,) —(B) 
Oz Cy 

It is obvious that unless = + 1.00 there is no single rela- 
tionship which holds for all pairs of facts. These equations 
express the best summary or “average” of the various re- 
lationships which exist. For Table XXIV, r =.85 + .01 
Oo, =47.9, 0, = 55.2, a, = 150.0 and a,= 154.5. If these 
values are substituted in equations B, we have: 


y’ = 980! +75 a!’ = .74y'+36 
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One interpretation of the equation y’ = .982'+ 7.5 is that 
for every unit of change in 2’ there is in general a change of 
98 unit iny’. Since in this illustration 2’ and y’ are in terms 
of comparable units (words per minute) this shows that a 
close relationship exists between them. If for each 2z-fact 
there is estimated a corresponding y-fact by means of the 
equation y’ = .982’ + 7.5, the coefficient of correlation of the 
estimated y-facts with the z-facts will be 1.00. This must 
be true because the pairs of facts lie on the straight line 
whose equation is y’ = .982’+ 7.5. Any set of y-facts esti- 
mated from the z-facts by means of an equation of the form 
y = mx +c would yield the same coefficient of correlation. 
The estimates made by means of the regression equation, 
y’ = .982' = 7.5, involve the least possible changes from the 
actual y-facts.__ Therefore, the difference between actual 
y’s and the estimated y’s furnishes an indication of their 
departure from perfect correlation. The equation 2’ = 
.74y’ + 36 may be used to obtain a set of estimated z-facts 
which will correlate perfectly with the y-facts. 

In Table XXVI are given the differences between the 
actual score and the estimated score for a number of pupils 
whose scores are recorded in Table XXIV. The score on 
Form 1 of the Courtis Silent Reading Test No. 2 is the z- 
fact, and the score on Form 3 the y-fact. By means of the 
equation 

y’ = 98x’ +7.5 


estimated scores on Form 3 have been calculated. These 
estimated Form 8 scores correlate perfectly with the Form 1 
scores. ‘The differences between the actual Form 8 scores 


1The sum of the squares of the differences between the estimated y’s 
and the actual y’s is a minimum when the estimation is made by means of 
o o 
the equation y = 7 its or (y’— ay) =r s (w' — az). (Yule, G. U., An 
z 
Introduction to the Theory of Statistics, pp. 171-72). 
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TABLE XXVI. SHOWING ACTUAL SCORES AND ESTIMATED 
FORM 3 SCORES ON THE COURTIS FORM 3 RATE SCORES ON 
THE COURTIS SILENT READING TESTS NO. 2 


Pum Score on Score on Est. score on 2 
i Form 1 Form 3 Form 3 Dif. 

+ a 
1 98 100 104 4 
2 38 34 45 ll 
3 130 126 135 9 
4 203 187 206 19 
5 98 83 104 1 
6 69 70 ves) 5 
qT 94 110 100 10 
8 150 148 155 7 
9 85 89 91 2 
10 97 93 103 10 
11 157 160 161 1 
12 106 107 lll 4 
13 148 162 153 9 
14 130 141 135 6 
15 128 152 133 19 
16 182 180 186 6 
17 157, 149 161 12 
18 116 130 121 9 
19 108 113 113 0 
20 93 127 99 28 
21 81 72 87 15 
22 124 120 129 9 
23 154 188 158 80 
24 182 198 186 12 
25 113 127 118 9 


and the estimated Form 8 scores are given in the last column 
of the table. These differences indicate how much the scores 
obtained from the two forms of the tests fall short of perfect 
correlation. For the first pupil, the difference is — 4. For 
the second, it is — 11. The differences vary. Many are rel- 
atively small, but others are large. They are about equally 
divided as to sign. They represent the departures from 
perfect correlation. 

The probable error of estimate. It is possible to describe 
these differences of estimated scores from the actual scores 
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without calculating them. If y is taken to represent the 


OG, . . . 
actual score, r—“z is the obtained score.! The difference 
oz 


is 
Co. 
y—r yz, 
oz 


If the correlation is rectilinear and the distributions are 
normal, these differences will form a normal distribution 
with the average at zero. Hence the standard deviation of 
the distribution of the differences or departures from per- 
fect correlation may be expressed as follows: 


1 G. 
(© 4)? = 2-1 a) 


> y? . Lary o,\2 Ly? 
Sem ee Op ee (2 a 

N nw N os G. N 
= 0,?—2r a,? Yigg 


By the same method we can obtain 


Cra, = % Vi—r? 


Since the probable error permits of a more convenient 
interpretation than the standard deviation, we generally 
use the following equations and obtain the probable error of 
estimate. 'These expressions give the median deviation of 
the departures from perfect correlation. 

1 In this proof z and y are taken as deviations from the true averages. 


The absolute values could be used but to do so would make the proof 
cumbersome. _ 
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P.E..ny, = 6745 0, Vy] — 72 
P.E gs, — 6745 oy V 1 —_— 72 : 


Apply these formulas to Table XXIV. 
| -P.E.nu, = (.6745) (47.9) V1 — 852 
= 16.79 
P.E.py = (6745) (55.2) V1 — 85? 
=19.35 


This means that half of the scores obtained from Form 3 
of this test failed of perfect correlation with the Form 1 
scores by 19.35. For any pupil selected at random from 
this group the chances are 1 to 1 that his Form 3 score would 
depart from perfect correlation with his Form 1 score by as 
much as 19.35 words per minute. A similar interpretation 
may be given for the departure of the Form 1 scores from 
perfect correlation with the Form 3 scores. Since the two 
forms are assumed to be equivalent, the differences of the 
two probable errors of estimate may be considered to be due 
to chance variations in the values of x and y. Hence, the 
average of the two determinations should be taken as the 
most representative of the Courtis Silent Reading Test 
No. 2. 

P.E.zq = 18.07 


The error of estimate represented graphically. Fig. 13 
illustrates the meaning of the error of estimate. The scat- 
ter diagram is for the scores for Form 1 and Form 2 of the 
Illinois General Intelligence Scale, when they were given 
to a group of fifth-grade pupils. The codrdinates of each 
dot represent the two scores of a pupil. The oblique line 
represents the regression equation, y = 4.92 + .80z. Per- 
fect correlation of the Form 1 scores with the Form 2 scores 
would be secured if gach dot in the figure were shifted ver- 


RELATIONS BETWEEN PAIRED FACTS 353 


tically so that it would fall upon this regression line. A few 
short vertical lines have been drawn in the figure to indicate 
the magnitude of the shifts required. Some are very small, 


r= 924.006 
C7, =26.40 O,=23.05 
AV.=61.6 AV.=54.2 


b,=1.05 b,=.80 
A kE4.69+1.08 
90 Y=4.93+,80X 


IpSe24ers6r! 482 60 S72 6s 967-108-4120 
FORM 2 

Fig. 13. CorrELATIon or Form 1 Scores with Form 2 Scores OF THE 

Iuumois GENERAL INTELLIGENCE ScaLE, Firtra GRADE 


others are relatively large. Some are positive, others are 
negative. The probable error of estimate is the median 
amount of shift that is required to secure perfect correlation. 
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Since we are interested in how nearly our paired facts 
approach perfect correlation, we may describe the condition 
which exists in terms of the departure from perfect corre- 
lation. From this point of view Fig. 13 represents the 
extent to which the Form 1 scores depart from perfect cor- 
relation with the Form 2 scores. The probable error of 
estimate is thus a measure of the departure from perfect 
correlation. 

Probable error of measurement. When the paired facts, 
whose relation is being studied, are the scores derived from 
two applications of the same test or of duplicate forms of a 
test, it is possible to derive a formula which gives us a 
measure of the departure of either set of scores from perfect 
correlation with a corresponding set of true scores. The 
derivation of this formula has been explained on page p. 214. 
It is as follows: 


P.E.y = 


Oi +09 PY 

4 
When one is studying the reliability of a test this formula is 
recommended in preference to the one for the probable error 
of estimate. 

Coefficients of probable errors of estimate and probable 
errors of measurement. By means of the formule given 
above for the probable error of estimate and the probable 
error of measurement, we may calculate for any value of 
12 the factors by which the standard deviation is to be mul- 
tiplied in obtaining these measures of departure from per- 
fect correlation. In Table XXVII, the coefficients of the 
probable error of estimate and the probable error of measure- 
ment are given for values of 7,2 ranging from .00 up to .99. 
To find either the probable error of estimate or the probable 
error of measurement for a given value of rj, it is only 
necessary to multiply the standard deviation by the ap- 
propriate coefficient. 


1 eon i 
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This table helps us to realize the significance of a coeffi- 
cient of correlation. A coefficient of correlation of .65 is 
considered high, and yet we find by means of this table that 


TABLE XXVII. COEFFICIENTS OF PROBABLE ERRORS OF 
ESTIMATE AND PROBABLE ERRORS OF MEASUREMENT 


Coefficient of Probable error Probable error 
correlation, rig of estimate of measurement 
-00 6745 6745 
-05 - 6736 6574 
-10 6711 - 6399 
15 - 6669 5544 
-20 6609 - 6033 
225 -6531 . 5841 
-30 6434 - 5644 
85 . 6306 5438 
.40 .6182 . 5225 
245 . 6023 . 5002 
-50 5841 -4769 
55 - 5634 -4525 
.60 . 5396 - 4266 
.65 . 5126 . 8990 
-70 4817 - 38694 
75 4461 - 3373 
-80 .4047 .3016 
85 8553 - 2612 
.90 . 2940 . 2132 
.92 . 2643 - 1907 
94 . 2301 - 1652 
-95 . 2106 .1508 
-96 - 1889 - 1349 
ats . 1638 .1168 
.98 . 1342 0952 
-99 0952 . 0674 


in the case of 50 per cent of the pairs of values, there is a de- 
parture from perfect correlation which equals or exceeds 
.5126 of the standard deviation of the set of measures being 
considered. In fact, for coefficients of correlation of .90 or 
higher, both the probable error of estimate and the proba- 
ble error of measurement are relatively large. Thus, from 
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the standpoint of departures from perfect correlation, it is 
clear that values of rjg which we have been accustomed to 
interpret as indicating a “high correlation” are really in- 
dicative of departures from perfect correlation so large that 
it becomes necessary to revise our concept of “high correla- 
tion.” 
Effect of selection of data upon coefficients of correlation. 
We have already noted that, since it is necessary in studying 
general relationships to use limited samples of data, it is 
necessary to take into account the probable error of the co- 
efficient of correlation due to sampling. In the case of data 
collected from school children it is necessary to bear in mind, 
in addition to the above, the type of the population from 
which the data are collected. We may collect data from 
pupils belonging to a single school grade, or to a series of 
school grades. In Table XXVIII coefficients of correlation 
for pupils in grades III-B to VIII-A of one school system are 
given. In every case, all children of the school system were 
tested. Each grade group is sufficiently large so that the 
probable error of the coefficient of correlation, due to sam- 
pling, is relatively small. In the third column, the coeffi- 
cients of correlation of mental age with chronological age 
are given. For each grade group they are negative. Sev- 
eral of them are sufficiently large negative quantities to in- 
dicate a distinct inverse relationship between chronological 
age and mental age within the population group from which 
these data were secured. This, of course, is in agreement 
with our general observations. The bright pupils in any 
grade are the younger ones. The older pupils in any grade 
are generally the dull ones. However, when all grades are 
combined, the coefficient of correlation of chronological age 
with mental age is .56, which indicates a distinct positive 
relationship between mental age and chronological age 
when all grades are taken together. This again is in accord 
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with our general observations. Hence, we have an illustra- 
tion which emphasizes that the coefficient of correlation is 
materially affected by the grade-range of the population 
group from which the data are collected. 


TABLE XXVIII. EFFECT OF NATURE OF TRAIT AND SELEC- 
TION OF POPULATION GROUP UPON COEFFICIENT OF 
CORRELATION 


Achievement Chronological Mental Age 


pet seg Age and Achieve- and Achieve- 
otrent 


Age ment Age 


In the next three columns the coefficients of correlation 
for intelligence quotients with achievement quotients are 
given. These coefficients are negative. In the case of 
arithmetic there are relatively large negative quantities. In 
fact, they are slightly larger than the ones for chronological 
age with mental age. When all grades are combined the co- 
efficients of correlation are all negative, and differ but little 
from the average of the coefficients of correlation for the 
separate grade groups. Here we have a distinct reversal of 
the condition illustrated in Column 3. Hence it appears 
that the coefficient of correlation is affected not only by the 
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grade-range of the population group, but also by the nature 
of the traits which are considered. 

In the case of both the intelligence quotient and the 
achievement quotient the average is practically the same 
for all grade groups. The standard deviations are also 
essentially the same. Not only are they the same for all 
grade groups, but when all grades are combined approxi- 
mately the same averages and standard deviations are se- 
cured. We are in this case dealing with traits which do not 
increase from grade to grade, but tend to remain essentially 
constant. In the case of chronological age and mental age 
the traits increased from grade to grade. This is the ex- 
planation of the apparent inconsistency between the results 
for the two groups of traits. 

In the remaining columns of the table other coefficients 
of correlation are given which further illustrate the fact 
that, in interpreting coefficients of correlation, it is necessary 
to bear in mind the character of the population group from 
which the data were collected, and also the nature of the 
traits measured... 
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EXERCISES 


Form appropriate correlation tables for the following sets of paired facts. 
Then calculate the coefficient of correlation and the probable error of esti- 
mate. 


1. Rates or Sment ReapING MEASURED BY Form 1 anp Form 8 oF 
Courtis Sizent Reapine Test No. 2 


Form1 | Form 3| Form 1} Form8\| Form1\| Form8\ Form1\| Form$ 
159 169 118 100 102 112 128 152 
113 127 106 96 118 121 1385 143 
146 130 203 187 154 188 98 100 
193 223 193 191 130 141 150 148 
188 198 193 213 141 121 148 139 


142 163 157 143 144 149 103 99 
175 181 153 163 146 120 98 83 
85 89 157 160 182 180 126 117 


93 127 108 113 157 149 173 387 
157 207 139 129 193 147 124 120 
205 186 174 162 120 113 120 123 
118 108 175 208 183 198 116 130 

17 80 151 148 69 70 148 152 
280 286 130 124 193 230 148 162 
160 162 134 148 214 223 193 177 


93 81 279 289 254 236 | 153 169 
247 241 81 72 97 93 206 174 
94 110 38 34 131 113 177 176 


2. Scores oN Pressey Primer Test anv InpIANA SCALE or ATTAIN@ 
MENT No. 1 


P.P. \Ind. Att.) P.P. |Ind. Att.) P.P. |Ind. Att} P.P. |Ind. At. 


— | ff | fe —  f Sf | 


65 21 59 21 62 14 29 9 
69 17 71 20 55 8 51 9 
50 11 27 20 61 18 43 12 
52 15 50 22 68 10 70 22 
50 22 52 20 34 19 6 4 
55 22 79 22 73 20 54 19 
60 21 45 14 36 10 53 14 
380 22 20 22 18 3 59 16 
13 22 30 21 57 21 54 6 
67 21 51 21 41 17 60 17 
69 14 56 6 38 11 8 8 
83 20 56 21 25 16 49 16 
65 16 39 AG; 43 17 19 6 
66 22 58 20 45 18 84 18 
82 15 63 21 18 4 41 12 
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3. CHRonoLoaicaL AGE AND INTELLIGENCE Scorn. GraprE 3A 


Chron. | Intelli- | Chron. | Intelli- | Chron. | Intelli- | Chron. | Intelli- 


age gence age gence age gence age gence 
score score score score 
9 28 9 24 10 32 10 12 
10 33 10 36 9 55 10 22 
9 31 9 27 10 25 10 35 
9 45 9 34 9 4A 9 44, 
9 38 15 27 11 43 9 52 
9 47 10 31 10 37 9 31 
9 48 9 28 10 49 9 40 
10 13 8 43 9 56 9 38 


Calculate the coefficient of correlation and the probable error of estimate 
for the following correlation tables. 


4. CoRRELATION TABLE FOR SCORES YIELDED By Smrirs 2A AND 
Srerizs 2B or Barr’s Diagnostic Test In AMERICAN History 


Series 2A 
Series 
2B | 
15| 20} 25 | 30 | 35 | 40| 45) 50] 55 | 60! 65] 70|75| 80/85) T 

80 1 1 
15 1 Pit 1 4 
70 2) SE ka) dehs 10 
65 Lies 6| 4 Sark 17 
60 ETS | CSG Site 1 21 
55 oi Vill) 66s a 20 
50 4| 4) 4] 11S 2 17 
45 Fleet os 2| 1 1 17 
40 1 3| 8 1 13 
35 g a 3 
30 1 1 1 1 4 
25 a 1 


| i 4 |17) 15/17/19) 21/15} 6)6|4/]21]1 | 198 
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5. CorRELATION TABLE FoR ScoRES YIELDED By Form 1 anp Form 2 
OF THE InLiINois GENERAL INTELLIGENCE SCALE 


Form 1 
Form 
5 | 10] 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 | 55 | 60/65 /T 

65 i ae es: 
60 1 1 2 
55 1 1 1 3 
50 23 5 
45 at 2 Bs Ye ag tea | 1 10 
40 2 1 4 3 3 18 
35 1 1 2 = 3 g 1 1 14 
30 8 1 4 
25 1 = 2 g 8 
20 8 1 4 
15 3 2 1 6 
10 2 il $3 

5 1 A 
4k te Gas bao OR Te Hel Ol Ons | Ot a) cael 


6. CorRRELATION TABLE FoR CHRONOLOGICAL AGE AND MentTAL AGE oF 
Srxta-GravE Purpits 


Chronological age 
Mental 


9 | 94 | 10 }103| 11/114] 12 |194] 13 }134| 14 |143] 15 |15}/ 16] T 


18 2 2 
17 
16 
15 2) ey A Sk 14 
i Cat fa Ah ie | 3] 4] 4 13 
13 1 | 3] 4/18] 6] 6] 4 1 39 
12 2) 4/10/27;16| 8) 4) 1] 1 75 
11 2 | 5] 9/26/21;14) 8; 5/ 1 91 
10 1] 3] 6| 30/27/10; 7| 9; 3 100 
9 3| 8/13)10; 8) 6 51 
8 Siero) LO ee dal eed 32 
7 BS Boo tag a 2 4 2 
6 1 7 
T 1 | 9 | 16) 41 J117| 95 | 54 | 42] 24) 10 426 
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tation of, 341. 
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Coefficient of reliability, 202. 
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Common unit, 147. 
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Difficulties, 61, 94, 119. 
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Educational quotient, 157. 

Efficiency, 44, 172, 264. 

Elliot, E. C., 28 

Errors, see constant errors and vari- 
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Rate test, 68, 107. 

Recognition examination, 285. 

Regression equation, 348. 

Reliability, 201. 

Remedial instruction, 244, 251. 

Requirements of test construction, 
64, 68 ff. 


Rugg, H. 0., 321. 


Sampling, effect of, 330, 345. 
Scale, 15, 20, 106, 133. 
Scaled test, 62, 73, 78, 89, 92, 118. 
Secrist, Horace, 343. 
Skewness, 308. 

Spiral test, 63, 74. 

Standard deviation, 325. 
Standard units, 17, 21. 
Starch, Daniel, 28, 127. 
Stone, C. W., 6. 

Study tests, 51. 

Subjective, 26. 


Tests, definition, 20; selection of, 
rast 

Testing by teacher, 237. 

Testing conditions, control of, 81 ff. 

Theisen, W. W., 196, 241. 

Thorndike, E. a 2, 4, 24, 26, 122, 
133, 186, 209, 216, 344. 

T-score, 151. 
Trabue, M. R., 138, 145. 

True score, 201. 

True-false examination, 283. 


Uniform tests, 62, 72, 113. 


Validity, 188, 194, 234, 
Validity of significance, 227. 


Van Wagenen, M. J., 125, 127, 
241. 

Variable errors, 198, 243, 3829, 
B44 


Whipple, G. M., 321. 
Wood, Ben D., 284. 
Woodworth, R.S., 211. 
Woody, Clifford, 127, 145. 


Yule, G. Udny, 209, 349. 


Zero point, 101, 146, 150, 
Zirbes, Laura, 246. 


adits y patina 


iy was 


palttss covery 
ao) eeu 


ee 


2G ee 


‘ere 


ame ry ot et arars 
a * ets aoe ANTS) 
seca. 


neh : 


RIVERSIDE 
TEXTBOOKS IN EDUCATION 


General Educational Theory 


PSYCHOLOGY FOR NORMAL SCHOOLS. 
By L. A. Averitt, Massachusetts State Normal School, Worcester. 
EXPERIMENTAL EDUCATION. 
By F. N. Freeman, University of Chicago. 
HOW CHILDREN LEARN. 
By F. N. FREEMAN. 
THE PSYCHOLOGY OF THE COMMON BRANCHES, 
By F. N. Freeman. 
THE PRE-SCHOOL CHILD. 
By ARNOLD GgsELL, Ph.D., M.D., Director Yale Psycho-Clinic, Professor of 
Child Hygiene, Yale University. 
DISCIPLINE AS A SCHOOL PROBLEM. 
By A. C. Perry, JR. 
AN INTRODUCTION TO EDUCATIONAL SOCIOLOGY, 
By W. R. Smiru, Kansas State Normal School. 
TRAINING FOR EFFECTIVE STUDY. 
By F. W. Tuomas, State Normal School, Fresno, California. 


AN INTRODUCTION TO CHILD PSYCHOLOGY. 
By C. W. Wapp kg, Ph.D., Los Angeles State Normal School 


History of Education 


THE HISTORY OF EDUCATION. 
By E. P. CuBBERLEy. 


A BRIEF HISTORY OF EDUCATION, 
By E. P. CupBer.ey. 


READINGS IN THE HISTORY OF EDUCATION, 
By E. P. CuBBer.ey. 


PUBLIC EDUCATION IN THE UNITED STATES. 
By E. P. Cuppgervey. 


Administration and Supervision of Schools 


HEALTHFUL SCHOOLS: HOW TO BUILD, EQUIP, AND MAINTAIN 
THEM. 
By May Ayrgs, J. F. Witttams, M.D., University of Cincinnati, and T. D. 
Woop, A.M., M.D., Teachers College, Columbia University. 
PUBLIC SCHOOL ADMINISTRATION, 
By E. P. Cuppervey. 
RURAL LIFE AND EDUCATION, 
By E. P. CuspBgervey. 
A GUIDE TO EDUCATIONAL MEASUREMENTS. 
By Haran C. Hines, Assistant Professor of Education, The University of 
Washington. 
HEALTH WORK IN THE SCHOOLS. 
By E. B. Hoac, M.D., and L. M. Terman, Leland Stanford Junior University. 
INTRODUCTION TO THE THEORY OF EDUCATIONAL MEASURE- 
MENTS. 
By W.S. Monrog, University of Illinois. 
MEASURING THE RESULTS OF TEACHING. 


By W. S. Monrog. 
1926 a 


EDUCATIONAL TESTS AND MEASUREMENTS. 
By W. S. Monroz, J. C. DeVoss, Kansas State Normal School; and F. j. 
Kg ty, University of Kansas. 


THE SUPERVISION OF INSTRUCTION. 
By H. W. Nutt, University of Kansas. 


STATISTICAL METHODS APPLIED TO EDUCATION. 
By H. O. Ruac, University of Chicago. 


CLASSROOM ORGANIZATION AND CONTROL, 
By J. B. Szars, Leland Stanford Junior University. 


A HANDBOOK FOR RURAL SCHOOL OFFICERS. 
By N. D. SHowatter, Washington State Normal School. 


THE HYGIENE OF THE SCHOOL CHILD. 
By L. M. Terman. 


THE MEASUREMENT OF INTELLIGENCE. 
By L. M. Terman. 
Test Material for the Measurement of Intelligence. Record Booklets for the 
Measurement of Intelligence. 


THE INTELLIGENCE OF SCHOOL CHILDREN. 
By L. M. Terman. 


Methods of Teaching 


TEACHING LITERATURE IN THE GRAMMAR GRADES AND HIGH 
SCHOOL. 
By Emma M. Boventus. 


HOW TO TEACH THE FUNDAMENTAL SUBJECTS. 
By C. N. Kennact and G. A. Mrrick. 


HOW TO TEACH THE SPECIAL SUBJECTS. 
By C. N. Kenpatt and G. A. Miricx. 


SILENT AND ORAL READING. 
By C. R. Strong. 


THE TEACHING OF SCIENCE IN THE ELEMENTARY SCHOOL, 
By G. H. Trarton, State Normal School, Mankato, Minnesota. 


TEACHING IN RURAL SCHOOLS. 
By T. J. Woortsr, University of Georgia. 


Secondary Education 


fHE JUNIOR HIGH SCHOOL. 
By Tuos. H. Brices, Columbia University. 


THE TEACHING OF ENGLISH IN THE SECONDARY SCHOOL 
By CHARLES Swain THOMAS. 


PRINCIPLES OF SECONDARY EDUCATION, 
By AvexanperR InGuis, Harvard University. 


PROBLEMS OF SECONDARY EDUCATION. 
By Davip SNeppzN, Columbia University. 


HOUGHTON MIFFLIN COMPANY 


1926b 


VOCATIONAL PREPARATION 


THE VOCATIONAL GUIDANCE OF YOUTH 
By Meyer Bloomfield 


A monograph by the former Director of the Vocation Bureau of Boston. 


YOUTH, SCHOOL, AND VOCATION By Meyer Bioomfielr 


A first-hand presentation of the meaning and work of the vocational guidance 
movement. ; 


CHOOSING A VOCATION By Frank Parsons 


This book is an indispensable manual for every vocational counselor. 


THE PROBLEM OF VOCATIONAL EDUCATION 
By David Snedden 


The author is the Professor of Education, Teachers College, and one of the 
leaders in the movement for the closer adaptation of public schools to the actual 
needs of youth. 


PREVOCATIONAL EDUCATION 
By Frank M. Leavitt and Edith Brown 


The first authoritative book to tell how the public schools may prepare pupils ta 
select wisely the work to which they are best adapted. 


THE PEOPLE’S SCHOOL By Ruth Mary Weeks 


~ siatenest regarding the vocational training movement in this country and 
abroad, 


VOCATIONS FOR GIRLS 
By Mary A. Laselle and Katherine Wiley 


Information as to conditions of work and the opportunities in the more common 
vocations open to girls with only a high-school education. 


VOCATIONAL EDUCATION 
By David Snedden, Ruth Mary Weeks, and Ellwood P. Cubberley 


A combination of three volumes from the Riverside Educational Monographs 
treating different phases of vocational education,—theory, administration, and 
practice. 


PRINCIPLES AND METHODS OF INDUSTRIAL 
EDUCATION By William H. Dooley 


This is a book for use in teacher being classes. There is an Introduction by 

Charles A. Prosser, and an equipment of thought stimulating questions, together 

with reading references and courses of study. 

INDUSTRIAL EDUCATION: Its Problems, Methods, 
and Dangers By Albert H. Leake 


A study and criticism of the opportunities provided for the education of the 
industrial worker. 


ESTABLISHING INDUSTRIAL SCHOOLS 
By Harry Bradley Smith 
A practical discussion of the steps to be taken in establishing industrial schools 


HOUGHTON MIFFLIN COMPANY 


08 


DATE DUE . 


ape 01g 04. 


FEB. 1 U 1986 


\ 


Se Se ee de 


7196 M7! BIBLE COLLEGE & SEM. LIBRARY 


371.26 M753i main 
| 320 0002 . 


uC’ 


ii 
2 4 


| 
3 4 


69 


SEMINARY LIBRARY 
271.246 MPSS . 
| Ac. #42374 
Monroe, Walter Seott 


An introduction ty the 
theory of educational 
measurements 


oenco 


| 
THE CINCINNATI BIBLE 


ay 
eirscs 


See 


Neck a er 
Re para ti 6 Pot 


seer 


